The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index; 5 of the 29 sources its claims cite were checked for support.
Demystifying LLM-Based Software Engineering Agents
Open accessAbstract
Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents? To attempt to answer this question, we build Agentless – an agentless approach to automatically resolve software development issues. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (32.00%, 96 correct fixes) and low cost ($0.70) compared with all existing open-source software agents at the time of paper submission! Agentless also achieves more than 50% solve rate when using Claude 3.5 Sonnet on the new SWE-bench Verified benchmark. In fact, Agentless has already been adopted by OpenAI as the go-to approach to showcase the real-world coding performance of both GPT-4o and the new o1 models; more recently, Agentless has also been used by DeepSeek to evaluate their newest DeepSeek V3 and R1 models . Furthermore, we manually classified the problems in SWE-bench Lite and found problems with exact ground truth patches or insufficient/misleading issue descriptions. As such, we construct SWE-bench Lite- S by excluding such problematic issues to perform more rigorous evaluation and comparison. Our work highlights the currently overlooked potential of a simplistic, cost-effective technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction. We have open-sourced Agentless at: https://github.com/OpenAutoCoder/Agentless
Overview
CoChat Summary
TL;DR
Agentless, a deliberately non-agentic three-phase pipeline for localization, repair, and patch validation, resolves 96/300 SWE-bench Lite issues (32.00%) at $0.70 average cost, the highest among open-source approaches at submission time. The paper also manually classifies SWE-bench Lite problems and constructs a filtered SWE-bench Lite-S benchmark.
Key findings
- On SWE-bench Lite, Agentless with GPT-4o resolves 96/300 problems (32.00%) at average $0.70 and 78,165 tokens; correct-location rates are 20.7% line, 50.3% function, and 69.7% file.
- On SWE-bench Verified, Agentless resolves 194/500 problems (38.80%) with GPT-4o and achieves more than 50% solve rate with Claude 3.5 Sonnet.
- Ablations show patch selection improves from 77 fixes with majority voting to 81 with regression tests and 96 with reproduction tests; repair with 4×10 multi-samples reaches 96 fixes versus 88 for greedy locations and 85 for merged multi-samples.
- Manual classification finds 4.3% of SWE-bench Lite issues contain the exact ground-truth patch, 9.7% contain complete solution steps, 5.0% contain misleading solutions, and 10.0% lack enough information; the filtered SWE-bench Lite-S has 249 problems, where Agentless resolves 84 (33.73%).
Methods
- Agentless uses hierarchical localization: top-N suspicious files from repository structure plus embedding retrieval, then classes/functions via skeleton format, then fine-grained edit locations; repair samples Search/Replace diff patches; patch validation generates reproduction tests and uses regression tests plus majority voting.
- Evaluation uses SWE-bench Lite (300 problems) against 26 agent-based baselines, with GPT-4o (gpt-4o-2024-05-13), text-embedding-3-small, top 3 files, 4 edit-location samples, 10 patches per sample (40 patches per issue), and 40 reproduction tests; metrics are % Resolved, average cost, average tokens, and correct location at line/function/file.
- The paper manually classifies SWE-bench Lite problems and constructs SWE-bench Lite-S (249 problems), and also reports results on SWE-bench Verified (500 problems).
Limitations
- Stated internal threat: possible data leakage because SWE-bench Lite ground-truth developer patches may be in GPT-4o’s training data; since GPT-4o is closed-source, this cannot be checked, though the authors note prior work used similar closed LLMs and SWE-bench authors found no significant pre/post-cutoff difference.
- Stated external threat: SWE-bench Lite may not generalize to other datasets, and it mostly focuses on bug-fixing issues; future work is planned on other benchmarks.
- Evident from the paper’s own results: reproduction tests are imperfect—213/300 tests reproduce the original issue, but only 94 correctly output “Issue resolved” after applying the ground-truth patch; issue descriptions can lack sufficient information, and Agentless underperforms closed-source agents when no location clue is provided.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 4 Not valid · 14 Not checked · 106 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“That is to say they tend to take all information/feedback and do not know how to filter out or correct irrelevant, incorrect, or misleading information [Shi et al. 2023;Zhang et al. 2023].”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
What the sources say
- Not matchedcites Shi et al. 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Zhang et al. 2023 sourceExperimental — accuracy not yet measured
Section 4, “Sources of LLM Hallucination” (page not available), says LLMs can overestimate their ability to assess their own answers and may favor a user’s perspective over accuracy. Section 5.4.2, “Resorting to External Knowledge” (page not available), calls verification of retrieved information an open challenge. These passages support difficulties with self-assessment and unreliable information, but not the blanket claim that agents take all feedback or cannot filter or correct it. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, we observed that SWE-bench Lite contains problems with exact ground truth patch in the description (4.3%), problems with missing critical information needed to solve the issue (10.0%), and problems that include misleading solutions in the issue description (5.0%).”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As such, we construct SWE-bench Lite-𝑆 by excluding such problematic issues to perform more rigorous evaluation and comparison.”
- Also stated in: § Abstract · plain
“Recognizing these issues, we built SWE-bench Lite-𝑆, which removes such problematic problems, and serves as a more rigorous benchmark to evaluate the ability to solve real-world software development challenges.”
- Also stated in: § Abstract · plain
“To address these issues, we constructed a filtered dataset of SWE-bench Lite-𝑆 that excludes such problematic issues, enabling more rigorous evaluation and comparison.”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Abstract · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, we observed that SWE-bench Lite contains problems with exact ground truth patch in the description (4.3%), problems with missing critical information needed to solve the issue (10.0%), and problems that include misleading solutions in the issue description (5.0%).”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, we observed that SWE-bench Lite contains problems with exact ground truth patch in the description (4.3%), problems with missing critical information needed to solve the issue (10.0%), and problems that include misleading solutions in the issue description (5.0%).”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
64 more claims
Create a free account to see all 129 claims and to verify the 4 references still open.
- Probably not a substantial claim. 28
- It may state two separate claims as one. 39
- Borderline: one of the tests for a substantial claim is unclear. 1
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unlike single-turn code completion, practical coding scenarios require an agent to sustain a sequence of interdependent steps across many rounds of interaction, such as requirement understanding, code localization, evidence retrieval, multi-file editing, test execution, bug fixing, and regression verification [176, 177].”
From Code as Agent Harness (2026) · cites this paper as [176]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Demystifying LLM-Based Software Engineering Agents > 5.2.2 Repair Ablation.
“Starting with using the greedy location set generated in the edit location generation stage, we observe that we can already achieve very high performance of more than 88 issues fixed.”
§ Demystifying LLM-Based Software Engineering Agents > 8 Conclusion
“| Greedy location (40 samples) | 88 (29.33%) | $0.22 |”
Citations
Cites
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Self-citationThe probabilistic relevance framework: BM25 and beyond
Robertson, S., & Zaragoza, H. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000019 · Cited by 3,374
Modern Information Retrieval : A Brief Overview
Singhal, A. (2001) · IEEE Data(base) Engineering Bulletin · Cited by 1,556
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Empirical evaluation of the tarantula automatic fault-localization technique
Jones, J. A., & Harrold, M. J. (2005) · Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/1101908.1101949 · Cited by 1,238 · cited 1×
“Dynamic FL techniques mainly include spectrum-based FL (SBFL) [Abreu et al. 2009, 2007; Jones and Harrold 2005] and mutation-based FL (MBFL) [Lou et al. 2020; Moon et al. 2014; Papadakis and Le Traon 2015].”
A Survey on Software Fault Localization
Wong, W. E., Gao, R., Li, Y., Abreu, R., & Wotawa, F. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2521368 · Cited by 1,110 · cited 1×
“Fault localization (FL) [Wong et al. 2016] techniques aim to identify the suspicious locations (e.g., statements or methods) in source code related to bugs.”
GenProg: A Generic Method for Automatic Software Repair
Le Goues, C., Nguyen, T., Forrest, S., & Weimer, W. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.104 · Cited by 1,085 · cited 1×
“Traditional APR techniques can be categorized as template-based [Ghanbari et al. 2019; Liu et al. 2019], heuristic-based [Le et al. 2016; Le Goues et al. 2012], and constraint-based [Long and Rinard 2015; Mechtaev et al. 2016] tools.”
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · cited 1×
“With the emergence and popularity of agent-based frameworks [Xi et al. 2023], recently researchers and industry practitioners have begun developing agent-based approaches to solve software engineering tasks.”
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
On the Accuracy of Spectrum-based Fault Localization
Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2007) · Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007) · doi:10.1109/taic.part.2007.13 · Cited by 736
Research works printed but not linked: 9
- 10.5555/1308173.1308264 · doi:10.5555/1308173.1308264
- Jie Huang Xinyun Chen Swaroop Mishra Huaixiu Steven Zheng Adams Wei Yu Xinying Song and Denny Zhou. 2024. Large Language Models Cannot Self-Correct ReasoningYet. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=IkmD3fKBPQ
- Carlos E Jimenez John Yang Alexander Wettig Shunyu Yao Kexin Pei Ofir Press and Karthik R Narasimhan. 2024b. SWE-bench Leaderboard. https://www.swebench.com/.
- Sophia D Kolak Ruben Martins Claire Le Goues and Vincent Josua Hellendoorn. 2022. Patch Generation with Language Models: Feasibility and Scaling Behavior. In Deep Learning for Code Workshop.
- Theo X Olausson Jeevana Priya Inala Chenglong Wang Jianfeng Gao and Armando Solar-Lezama. 2023. Is Self-Repair a Silver Bullet for Code Generation?. In The Twelfth International Conference on Learning Representations.
- Xianfei Ou Cong Li Yanyan Jiang and Chang Xu. 2024. The Mutators Reloaded: Fuzzing Compilers with Large Language Model Generated Mutation Operators. In ASPLOS.
- Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning. PMLR, 31210–31227.
- Chunqiu Steven Xia Matteo Paltenghi Jia Le Tian Michael Pradel and Lingming Zhang. 2024. Universal Fuzzing via Large Language Models. In 46th International Conference on Software Engineering (ICSE).
- Chunqiu Steven Xia Yuxiang Wei and Lingming Zhang. 2023b. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery.
Cited by
Items 101–129 of 129
Historical Artifacts Do Not Measurably Improve Knowledge Graph-Augmented Issue Resolution
Heisler, J., & Fischer, A. (2026) · International Conference on Advanced Computer Information Technologies (ACIT) · doi:10.1109/acit69831.2026.11700062
“Although KGCompass and analogous tools [4], [5], [6], [7] have been demonstrated to exhibit superior performance in comparison to baselines, no previous studies have systematically isolated the contribution of the historical issue layer.”
The Developer as Curator: A Canvas for Coding Agent Context
Böhme, L., Beckmann, T., Perscheid, M., & Hirschfeld, R. (2026) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3840586.3843212
SWE-PDB: Teaching LLMs to Leverage Debugging Tools via Agentic Training
Liu, J., Hu, X., & Xia, X. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832117 · Cited by 1
“Existing approaches in this setting can be broadly grouped into procedure-based methods [17, 32, 53, 57] and agent-based methods [6, 50, 60, 69].”
On-Device GitHub Contributions Audit utilizing Recursive Language Models
Effendy, C. L., Djayadi, D., Wati, P. M., & Saputri, T. R. D. (2026) · International Conference on Technology Innovation and Its Applications (ICTIIA) · doi:10.1109/ictiia70925.2026.11717373
Cost-Effective Repository Exploration for Agentic Issue Localization
Awad, M. N. A., & Ivanov, S. (2026) · International Workshop on Agentic AI for Next-Generation Software Development · doi:10.1145/3843282.3844422 · Cited by 1
To Think or Not to Think: Evaluating LLM Reasoning and Agents in Vulnerability Detection
Tan, H. B., Widyasari, R., Astekin, M., Göknil, A., Sözer, H., Tun, Y. N., Husom, E. J., Shar, L. K., & Jiang, L. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837521
Who Should Own the Loop? Harness Decomposition for Small-Model Repository Repair
Dente, F., Satriani, D., Santoro, D., Veltri, E., & Papotti, P. (2026) · International Workshop on Agentic AI for Next-Generation Software Development · doi:10.1145/3843282.3844442
How Quantum Bugs Live and Die: A Lifecycle-Based Empirical Study of Bugs in Quantum Software
Shi, Y., Meng, X., Huang, X., Zhang, J., Wo, T., & Wang, X. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834439 · Cited by 1
Wan, Y., Liang, T., Xu, J., Xiao, J., Huo, Y., & Lyu, M. R.-T. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834388
Command Coding: A Methodological Framework for AI-Accelerated Software Development
Villani, T. S., & Kellogg, M. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834536
“Systems likeMetaGPT[13] and ChatDev [24] assign distinct roles (product manager, architect, developer, tester) to specialized agents that collaborate through structured communication protocols, though more recent work has questioned the need for such complexity [34].”
From Automated Program Repair to Automated Agent Repair: Why Failed Trajectories Matter
Zhao, X. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3844579
Failure Explanations: The Missing Abstraction between Debugging and Repair
Smytzek, M. (2026) · IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1145/3843776.3844670
CiL-Agent: Compiler-in-the-Loop LLM Agent for Memory-Safe C Program Repair
Shen, Y., Zhang, Y., Yao, J., Liu, M., & Dong, W. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837493
Catching Developers in the Flow: Low-Latency Agentic Program Repair at Google Scale
Ziftci, C., Greene, S., Liu, R., Dalloro, L., & Dini, L. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834511
“Agentless [70] employs generated tests and a pre-defined three phase repair pipeline comprising of localization, repair and validation, achieving state-of-the-art performance at a remarkably low cost, outperforming previous open-source techniques.”
Schema-Conditioned Next-Tool Prediction for Coding Agents under Interface Drift
Nyberg, S., & Karlsson, I. (2026) · International Workshop on Agentic AI for Next-Generation Software Development · doi:10.1145/3843282.3844430
Santos, R. E. S., Santos, Í., & de Magalhães, C. V. C. (2026) · doi:10.1145/3843782.3844651
Fang, S., Ding, W., Zhang, M., Chen, Z., & Xu, B. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3850240 · Cited by 4
“5 demonstrates severe vulnerability to structural attacks (performance drops from 64.6% to 1.1”
InfluentialRanking Is Not Authority: Executable Context-Routing Contracts for Coding Agents
Mahesh, N. (2026) · Proceedings of the 1st International Workshop on PromptOps and Vibe Coding · doi:10.1145/3843779.3844639
Let’s Chat about Design: A Transformation-Oriented Viewpoint for LLM-Native Software
Braberman, V. A., & Bonomo, F. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834545
AnchorRepair: A Reference-Guided Dual-Agent Framework for Code Translation Repair
Sun, J., Zhao, Y., Li, F., & Jiang, J. (2026) · IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1145/3843776.3844669
“The rise of agent-based frameworks [21, 32] has produced significant interest in applying LLM agents to software engineering tasks [33, 39, 44].”
SecChain: Operationalizing Security Guidelines for Multi-agent Smart Contract Generation
Sun, D., Nie, L., & Liu, Y. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837480
“For secure code generation in general-purpose languages, prior work injects security knowledge, critics, or review-time signals into code generation and analysis, as in CodeGuarder [15], Secure-Coder [43], SecureReviewer [17], and INDICT [14].”
SE-Blackboard: A Shared-State Architecture for Multi-Agent Software Engineering Pipelines
Liu, E., Zhu, Q., & Dong, X. (2026) · IEEE Access · doi:10.1109/access.2026.3695435
Lindenbauer, T., Slinko, I., Felder, L., Bogomolov, E., & Zharov, Y. (2025) · arXiv · doi:10.48550/arxiv.2508.21433
“…improving the effectiveness of SE agents by scaling training data [10, 22, 36], selecting the most promising of multiple attempts [22, 10, 40, 2, 1, 33], providing execution-free or execution-based feedback to the agent through critics [2, 22, 10, 40, 24], or enhancing the agent’s planning…”
When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation
Yang, H., Yu, Q., & Desell, T. (2026) · arXiv · doi:10.48550/arxiv.2607.10569
“Engineered surfaces continue to be explored on the other end, layering AST-aware retrieval, semantic search, or fixed localize–repair–validate pipelines on the SWE-agent line [26, 39, 46].”
ReproAgent: Contract-Guided Paper-to-Code Reproduction
Hu, X., Pan, Z., Wang, Z., Liu, Z., Su, Z., & Zhang, W. (2026) · arXiv · doi:10.48550/arxiv.2608.24291
“…(Baek et al., 2024; Lu et al., 2024; Schmidgall et al., 2025), ML engineering (Chan et al., 2024; Tang et al., 2024), and code-generation or repository-level coding settings (Qian et al., 2024; Islam et al., 2024; Hong et al., 2024; Yang et al., 2024; Xia et al., 2025; Zhang et al., 2024).”
Ning, X., Tieu, K., Fu, D., Wei, T., Li, Z., Bei, Y., Zou, J., Ai, M., Liu, Z., Li, T.-W., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.18747
Self-citationReducing Token Usage of State-in-Context Agents using Minification
Hrubec, N., & Cito, J. (2026) · arXiv · doi:10.48550/arxiv.2606.01326
SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery
Virk, Y., Edds, Z., Xia, C. S., & Zhang, L. (2026) · arXiv · doi:10.48550/arxiv.2607.02807
Self-citation
Metadata
- Authors
- Xia, C. S., Deng, Y., Dunn, S., & Zhang, L.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Venue
- Proceedings of the ACM on Software Engineering
- Published
- 2025-06-19
- Volume
- 2
- Issue
- FSE
- Pages
- 801-824
- ISSN
- 2994-970X (ISSN-L)
- OpenAlex
- W4411449940
- Cited by
- 849
- References
- 34
- Citations by year
- 2026: 617 · 2025: 206
- FWCI
- 490.90
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the ACM on software engineering. (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- NSF (CCF-2131943)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- autonomous software agents, large language models, program repair, software development automation, SWE-bench Verified, patch validation, Claude 3.5 Sonnet, GPT-4o, DeepSeek-V3
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.