CoChat
Sign Up
Cited by 34Software Engineering ResearchSoftware Engineering Techniques and PracticesOpen Source Software Innovations

On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub

Open access
Watanabe, Miku, Li, Hao, Kashiwa, Yutaro, Reid, Brittany, Iida, Hajimu, Hassan, Ahmed E.·2026·ACM Transactions on Software Engineering and Methodology
DOI 10.1145/3798166
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) are increasingly being integrated into software development processes. The ability to generate code and submit pull requests with minimal human intervention, through the use of autonomous AI agents, is poised to become a standard practice. However, little is known about the practical usefulness of these pull requests and the extent to which their contributions are accepted in real-world projects. In this article, we empirically study 567 GitHub pull requests (PRs) generated using Claude Code, an agentic coding tool, across 157 diverse open source projects. Our analysis reveals that developers tend to rely on agents for tasks such as refactoring, documentation, and testing. The results indicate that 83.8% of these agent-assisted PRs are eventually accepted and merged by project maintainers, with 54.9% of the merged PRs being integrated without further modification. The remaining 45.1% require additional changes and benefit from human revisions, especially for bug fixes, documentation, and adherence to project-specific standards. These findings suggest that while agent-assisted PRs are largely acceptable, they still benefit from human oversight and refinement.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3798166

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Once code is placed inside the agent loop, the model must not only generate text, but also search repositories, edit code, execute tests, call APIs, query documentation, and verify intermediate results.

    “Once code is placed inside the agent loop, the model must not only generate text, but also search repositories, edit code, execute tests, call APIs, query documentation, and verify intermediate results [218, 219].”

    From Code as Agent Harness (2026) · cites this paper as [218]

    The Motivating Example (p. 3) describes Claude Code editing files, searching for documentation, executing tests, and using tools connected to filesystems, command-line interfaces, and cloud services. It does not establish that agents must perform every listed action, or specifically say they call APIs and verify intermediate results. The suggested checks in Implications for coding agent builders (p. 18) do not fill those gaps. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Contemporary LLMs are able to generate pull requests of substantial size, including generating full files or making simultaneous nontrivial changes in multiple places in the code base.

    “Given that contemporary LLMs are able to generate pull requests of substantial size, including generating full files or making simultaneous nontrivial changes in multiple places in the code base [38], [39], this finding is concerning.”

    From Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code (2026) · cites this paper as [38]

    Section 4.1 (p. 8) reports that agent-generated PRs changed a median of two files and added more lines than human PRs; Section 5.2 (p. 17) says agents can produce extensive, multi-purpose changes. This supports substantial PRs and changes across multiple places, but our copy does not establish the specific claim that they generate full files. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 43

  1. [1][1] Adam Alami, Victor Vadmand Jensen, and Neil A. Ernst. 2025. Accountability in Code Review: The Role of Intrinsic Drivers and the Impact of LLMs. ACM Trans. Softw. Eng. Methodol. (2025). -
  2. [2][2] Md Mustakim Billah, Palash Ranjan Roy, Zadia Codabux, and Banani Roy. 2024. Are Large Language Models a Threat to Programming Platforms? An Exploratory Study. In Proceedings of the 18th IEEE/ACM International Symposium on Empirical Software Engineering and Measurement (ESEM'24) . 292-301. https://doi.org/10.1145/3674805.3686689 - · DOI
  3. [3][3] Lili Bo, Wangjie Ji, Xiaobing Sun, Ting Zhang, Xiaoxue Wu, and Ying Wei. 2024. ChatBR: Automated assessment and improvement of bug report quality using ChatGPT. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE'24) . 1472-1483. https://doi.org/10.1145/3691620.3695518 - · DOI
  4. [4][4] Islem Bouzenia, Premkumar T. Devanbu, and Michael Pradel. 2025. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE'25) . 2188-2200. -
  5. [5][5] Moataz Chouchen, Narjes Bessghaier, Mahi Begoug, Ali Ouni, Eman Abdullah AlOmar, and Mohamed Wiem Mkaouer. 2024. How Do So ware Developers Use ChatGPT? An Exploratory Study on GitHub Pull Requests. In Proceedings of the 21st IEEE/ACM International Conference on Mining Software Repositories (MSR'24) . 212-216. https://doi.org/10.1145/ 3643991.3645084 -
  6. [6][6] Mariana Coutinho, Lorena Marques, Anderson Santos, Márcio Dahia, César França, and Ronnie de Souza Santos. 2024. The Role of Generative AI in Software Development Productivity: A Pilot Case Study. In Proceedings of the 1st IEEE/ACM International Conference on AI-Powered Software (AIware'24) . 15-16. https://doi.org/10.1145/3664646.3664773 - · DOI
  7. [7][7] Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Zhen Ming (Jack) Jiang. 2023. GitHub Copilot AI pair programmer: Asset or Liability? J. Syst. Softw. 203 (2023), 111734. https://doi.org/10.1016/J.JSS.2023.111734 - · DOI · Paper page
  8. [10][10] Lishui Fan, Jiakun Liu, Zhongxin Liu, David Lo, Xin Xia, and Shanping Li. 2025. Exploring the Capabilities of LLMs for Code-Change-Related Tasks. ACM Trans. Softw. Eng. Methodol. 34, 6 (2025), 159:1-159:36. -
  9. [11][11] Xiaodong Gu, Meng Chen, Yalan Lin, Yuhan Hu, Hongyu Zhang, Chengcheng Wan, Zhao Wei, Yong Xu, and Juhong Wang. 2025. On the Effectiveness of Large Language Models in Domain-Specific Code Generation. ACM Trans. Softw. Eng. Methodol. 34, 3 (2025), 78:1-78:22. -
  10. [12][12] Qi Guo, Junming Cao, Xiaofei Xie, Shangqing Liu, Xiaohong Li, Bihuan Chen, and Xin Peng. 2024. Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE'24) . 34:1-34:13. -
  11. [13][13] Qi Guo, Xiaofei Xie, Shangqing Liu, Ming Hu, Xiaohong Li, and Lei Bu. 2025. Intention is All you Need: Refining your Code from your Intention. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE'25) . 1127-1139. -
  12. [14][14] Mohammed Mehedi Hasan, Hao Li, Emad Fallahzadeh, Gopi Krishnan Rajbahadur, Bram Adams, and Ahmed E. Hassan. 2025. Model Context Protocol (MCP) at First Glance: Studying the Security and Maintainability of MCP Servers. CoRR abs/2506.13538 (2025). - · arXiv
  13. [15][15] Ahmed E. Hassan, Gustavo Ansaldi Oliva, Dayi Lin, Boyuan Chen, and Zhen Ming Jiang. 2024. Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmap. CoRR abs/2410.06107 (2024). https://doi.org/10.485 50/ARXIV.2410.06107 - · arXiv
  14. [17][17] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. 33, 8 (2024), 220:1-220:79. https://doi.org/10.1145/3695988 - · DOI · Paper page
  15. [18][18] Dong Huang, Jie M. Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, and Heming Cui. 2025. Bias Testing and Mitigation in LLM-based Code Generation. ACM Trans. Softw. Eng. Methodol. (2025). -
  16. [19][19] Ivana Clairine Irsan, Ting Zhang, Ferdian Thung, David Lo, and Lingxiao Jiang. 2022. AutoPRTitle: A Tool for Automatic Pull Request Title Generation. In Proceedings of the 38th IEEE International Conference on Software Maintenance and Evolution (ICSME'22) . 454-458. https://doi.org/10.1109/ICSME55016.2022.00058 - · DOI
  17. [20][20] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In Proceedings of the 12th International Conference on Learning Representations (ICLR'24) . 7-11. -
  18. [21][21] Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction. In Proceedings of the 45th IEEE/ACM International Conference on Software Engineering, (ICSE'23) . 2312-2323. https://doi.org/10.1109/ICSE48619.2023.00194 - · DOI · Paper page
  19. [22][22] Stefano Lambiase, Gemma Catolino, Fabio Palomba, and Filomena Ferrucci. 2025. Motivations, Challenges, Best Practices, and Benefits for Bots and Conversational Agents in Software Engineering: A Multivocal Literature Review. ACM Comput. Surv. 57, 4 (2025), 93:1-93:37. https://doi.org/10.1145/3704806 - · DOI
  20. [23][23] J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33 (1977), 159-174. -
  21. [24][24] Feng Lin, Dong Jae Kim, and Tse-Hsun Chen. 2025. SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE'25) . 1527-1539. -
  22. [25][25] Chao Liu, Xindong Zhang, Hongyu Zhang, Zhiyuan Wan, Zhan Huang, and Meng Yan. 2024. An Empirical Study of Code Search in Intelligent Coding Assistant: Perceptions, Expectations, and Directions. In Proceedings of the 32th IEEE/ACM International Conference on the Foundations of Software Engineering (FSE'24) . 283-293. https://doi.org/10.1 145/3663529.3663848 -
  23. [26][26] Zhijie Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, and Liang Feng Zhang. 2024. No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT. IEEE Trans. Software Eng. 50, 6 (2024), 1548-1584. https: //doi.org/10.1109/TSE.2024.3392499 - · DOI · Paper page
  24. [27][27] Michael R. Lyu, Baishakhi Ray, Abhik Roychoudhury, Shin Hwei Tan, and Patanamon Thongtanunam. 2025. Automatic Programming: Large Language Models and Beyond. ACM Trans. Softw. Eng. Methodol. 34, 5 (2025), 140:1-140:33. -
  25. [28][28] Moritz Mock, Jorge Melegati, and Barbara Russo. 2024. Generative AI for Test Driven Development: Preliminary Results. In Proceeding of the 25th Software Engineering and Extreme Programming Workshops (XP'24) . 24-32. https: //doi.org/10.1007/978-3-031-72781-8\_3 - · DOI
  26. [29][29] Nadia Nahar, Haoran Zhang, Grace A. Lewis, Shurui Zhou, and Christian Kästner. 2025. The Product Beyond the Model - An Empirical Study of Repositories of Open-Source ML Products. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE'25) . 1540-1552. -
  27. [30][30] Changan Niu, Chuanyi Li, Vincent Ng, Dongxiao Chen, Jidong Ge, and Bin Luo. 2023. An Empirical Comparison of Pre-Trained Models of Source Code. In Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE'23) . 2136-2148. -
  28. [31][31] Changan Niu, Ting Zhang, Chuanyi Li, Bin Luo, and Vincent Ng. 2024. On Evaluating the Efficiency of Source Code Generated by LLMs. In Proceedings of the 1th IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE'24) . 103-107. https://doi.org/10.1145/3650105.3652295 - · DOI
  29. [32][32] David O'Brien, Sumon Biswas, Sayem Mohammad Imtiaz, Rabe Abdalkareem, Emad Shihab, and Hridesh Rajan. 2024. Are Prompt Engineering and TODO Comments Friends or Foes? An Evaluation on GitHub Copilot. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE'24) . 219:1-219:13. -
  30. [33][33] Jevgenija Pantiuchina, Bin Lin, Fiorella Zampetti, Massimiliano Di Penta, Michele Lanza, and Gabriele Bavota. 2022. Why Do Developers Reject Refactorings in Open-Source Projects? ACM Trans. Softw. Eng. Methodol. 31, 2 (2022), 23:1-23:23. -
  31. [34][34] David Piorkowski, Soya Park, April Yi Wang, Dakuo Wang, Michael J. Muller, and Felix Portnoy. 2021. How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study. Proc. ACM Hum. Comput. Interact. 5 (2021), 131:1-131:25. -
  32. [35][35] Tajmilur Rahman, Yuecai Zhu, Lamyea Maha, Chanchal Roy, Banani Roy, and Kevin A. Schneider. 2024. Take Loads Off Your Developers: Automated User Story Generation using Large Language Model. In Proceedings of the 40th IEEE International Conference on Software Maintenance and Evolution (ICSME'24) . 791-801. https://doi.org/10.1109/ICSME5 8944.2024.00082 - · DOI
  33. [36][36] Robson Santos, Ítalo Santos, Cleyton V. C. de Magalhães, and Ronnie de Souza Santos. 2024. Are We Testing or Being Tested? Exploring the Practical Applications of Large Language Models in Software Testing. In Proceedings of 17th IEEE International Conference on Software Testing, Verification and Validation (ICST'24) . 353-360. https: //doi.org/10.1109/ICST60714.2024.00039 - · DOI
  34. [37][37] Jiho Shin, Moshi Wei, Junjie Wang, Lin Shi, and Song Wang. 2024. The Good, the Bad, and the Missing: Neural Code Generation for Machine Learning Tasks. ACM Trans. Softw. Eng. Methodol. 33, 2 (2024), 51:1-51:24. -
  35. [38][38] Rosalia Tufano, Antonio Mastropaolo, Federica Pepe, Ozren Dabic, Massimiliano Di Penta, and Gabriele Bavota. 2024. Unveiling ChatGPT's Usage in Open Source Projects: A Mining-based Study. In Proceedings of the 21st IEEE/ACM International Conference on Mining Software Repositories (MSR'24) . 571-583. -
  36. [39][39] Rosalia Tufano, Luca Pascarella, Michele Tufano, Denys Poshyvanyk, and Gabriele Bavota. 2021. Towards Automating Code Review Activities. In Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering (ICSE'21) . 163-174. -
  37. [40][40] Zhijie Wang, Zijie Zhou, Da Song, Yuheng Huang, Shengmai Chen, Lei Ma, and Tianyi Zhang. 2025. Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE'25) . 2587-2599. -
  38. [41][41] Miku Watanabe, Yutaro Kashiwa, Bin Lin, Toshiki Hirao, Ken-ichi Yamaguchi, and Hajimu Iida. 2024. On the Use of ChatGPT for Code Review: Do Developers Like Reviews By ChatGPT?. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (EASE'24) . 375-380. https://doi.org/10.1145/3661167.3661183 - · DOI
  39. [42][42] Peter Weißgerber, Daniel Neu, and Stephan Diehl. 2008. Small patches get in!. In Proceedings of the 2008 International Working Conference on Mining Software Repositories (MSR'08) . 67-76. -
  40. [43][43] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated Program Repair in the Era of Large Pre-trained Language Models. In Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE'23) . 1482-1494. -
  41. [44][44] Xiao Yu, Lei Liu, Xing Hu, Jacky Wai Keung, Jin Liu, and Xin Xia. 2024. Fight Fire With Fire: How Much Can We Trust ChatGPT on Source Code-Related Tasks? IEEE Trans. Software Eng. 50, 12 (2024), 3435-3453. https: //doi.org/10.1109/TSE.2024.3492204 - · DOI
  42. [45][45] Qunhong Zeng, Yuxia Zhang, Zhiqing Qiu, and Hui Liu. 2025. A First Look at Conventional Commits Classification. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE'25) . 2277-2289. -
  43. [46][46] Sai Zhang, Zhenchang Xing, Ronghui Guo, Fangzhou Xu, Lei Chen, Zhaoyuan Zhang, Xiaowang Zhang, Zhiyong Feng, and Zhiqiang Zhuang. 2025. Empowering Agile-Based Generative Software Development through Human-AI Teamwork. ACM Trans. Softw. Eng. Methodol. 34, 6 (2025), 156:1-156:46.

Cited by

Sorted by citations

Authors
Watanabe, M., Li, H., Kashiwa, Y., Reid, B., Iida, H., & Hassan, A. E.
Affiliations
  • Nara Institute of Science and Technology (JP)
  • Queen's University (CA)
ORCID
Venue
ACM Transactions on Software Engineering and Methodology
Publisher
Association for Computing Machinery (ACM)
Published
2026-09-16
Volume
35
Issue
10
Pages
1-24
ISSN
  • 1049-331X (ISSN-L)
  • 1557-7392
DOI
10.1145/3798166
OpenAlex
W7134077757
Cited by
34
References
3
Citations by year
2026: 34
FWCI
284.96
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Transactions on Software Engineering and Methodology (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Japan Society for the Promotion of Science (JSPS) KAKENHI (Grants JP24K02921 and JP25K21359)
  • JST Precursory Research for Embryonic Science and Technology (Grant JPMJPR22P3, ASPIRE Grant JPMJAP2415)
  • AIP Network Laboratory Accelerated Program (JPMJCR25U7)
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
Keywords
agentic coding, large language models, GitHub pull requests, Claude Code, AI-assisted software development, pull request acceptance, human oversight, refactoring, documentation, testing, open source projects, code generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review