Why Are AI Agent–Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study
Open accessAbstract
Autonomous coding agents (e.g., OpenAI Codex, Devin, GitHub Copilot) are increasingly used to generate fix-related pull requests (PRs) in real-world software repositories. However, their practical effectiveness depends on whether project maintainers accept and merge these contributions. In this paper, we present an empirical study of AI agent–involved fix-related PRs, examining both their integration outcomes, latency, and the factors that hinder successful merging. We first analyze 8,106 fix-related PRs authored by five widely used AI coding agents from the AIDEV-POP dataset to quantify the proportions of PRs that are merged, closed without merging, or remain open. We then conduct a manual analysis of a statistically significant sample of 326 closed but unmerged PRs, spending approximately 100 person-hours to construct a structured catalog of 12 failure reasons. Our results indicate that test case failures and prior resolution of the same issues by other PRs are the most common causes of non-integration, whereas build or deployment failures are comparatively rare. Overall, our findings expose key limitations of current AI coding agents in real-world settings and highlight directions for their further improvement and for more effective human-AI collaboration in software maintenance.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Expectations, outcomes, and challenges of modern code review
Bacchelli, A., & Bird, C. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606617 · Cited by 650 · see citing works
An Analysis of the Automatic Bug Fixing Performance of ChatGPT
Sobania, D., Briesch, M., Hanna, C., & Petke, J. (2023) · 2023 IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1109/apr59189.2023.00012 · Cited by 338 · see citing works
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023) · arXiv · doi:10.48550/arxiv.2302.06590 · Cited by 287 · see citing works
Interval estimation for Cohen's kappa as a measure of agreement
Blackman, N. J., & Koval, J. J. (2000) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(20000315)19:5 · Cited by 219 · see citing works
Monperrus, M. (2014) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/2568225.2568324 · Cited by 175 · see citing works
Contemporary Peer Review in Action: Lessons from Open Source Development
Rigby, P. C., Cleary, B., Painchaud, F., et al. (2012) · IEEE Software · doi:10.1109/ms.2012.24 · Cited by 135 · see citing works
Li, H., Zhang, H., & Hassan, A. E. (2025) · arXiv (Cornell University) · doi:10.4230/lipics.esem.2026.67 · Cited by 78 · see citing works
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72 · see citing works
Agents in software engineering: survey, landscape, and vision
Wang, Y., Zhong, W., Huang, Y., Shi, E., Yang, M., Chen, J., Li, H., Ma, Y., Wang, Q., & Zheng, Z. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00544-2 · Cited by 50 · see citing works
On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
Watanabe, M., Li, H., Kashiwa, Y., Reid, B., Iida, H., & Hassan, A. E. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3798166 · Cited by 34 · see citing works
Deployed software component testing using dynamic validation agents
Grundy, J., Ding, G., & Hosking, J. (2004) · Journal of Systems and Software · doi:10.1016/j.jss.2003.05.001 · Cited by 24 · see citing works
A Systematic Literature Review on Large Language Models for Automated Program Repair
Zhang, Q., Fang, C., Xie, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.01466 · Cited by 17 · see citing works
Automatic prediction of rejected edits in Stack Overflow
Mondal, S., Uddin, G., & Roy, C. K. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10242-2 · Cited by 8 · see citing works
Self-citationMarsCode Agent: AI-native Automated Bug Fixing
Liu, Y., Gao, P., Wang, X., Liu, J., Shi, Y., Zhang, Z., & Peng, C. (2024) · arXiv · doi:10.48550/arxiv.2409.00899 · Cited by 5 · see citing works
Evaluating the Repair Ability of LLM Under Different Prompt Settings
Tian, X. (2024) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse62657.2024.00053 · Cited by 4 · see citing works
AI in Software Engineering: Perceived Roles and Their Impact on Adoption
Zakharov, I., Koshchenko, E., & Sergeyuk, A. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3730563 · Cited by 4 · see citing works
Random, A. (2025) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.18101034 · Cited by 1 · see citing works
What Drives Issue Resolution Speed? An Empirical Study of Scientific Workflow Systems on GitHubPreprint
Alam, K., & Roy, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.18852
Self-citationLi, H., Zhang, H., & Hassan, A. E. (2025) · arXiv · doi:10.48550/arxiv.2507.15003
Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmap
Hassan, A. E., Oliva, G. A., Lin, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.06107
Flagged first, then by citations
Research works printed but not linked: 1
- Shaowei Wang Tse-Hsun (Peter) Chen and Ahmed E. Hassan. 2018. How Do Users Revise Answers on Technical Q&A Websites? A Case Study on Stack Overflow. IEEE Transactions in Software Enginering (2018) 19.
Cited by
Still being checked.
- Authors
- Alam, K., Mondal, S., & Roy, B.
- Affiliations
- University of Saskatchewan (CA)
- Venue
- Proceedings of the 23rd International Conference on Mining Software Repositories
- Publisher
- ACM
- Published
- 2026-04-13
- Pages
- 964-968
- OpenAlex
- W7171968711
- Cited by
- 0
- References
- 21
- FWCI
- 0.00
- Type
- conference paper
- Open access
- hybrid
- doi.org
- Locations
- International Conference on Mining Software Repositories (MSR) (publishedVersion, cc-by)
- Funders
- This research is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grants program, and by the industry-stream NSERC CREATE in Software Analytics Research (SOAR). (410521)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- coding agents, autonomous coding agents, human-AI collaboration, software maintenance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
