Do AI-Generated Pull Requests Get Rejected More? (Yes but Why?)
Open accessAbstract
This paper investigates an important question in the era where AI can autonomously generate code and create pull requests: do AI-generated pull requests get rejected more than human-written ones and why do they get rejected? This investigation will help us understand how AI can support the open source community and its limitations. We conduct an in-depth examination of 1,779 rejected PRs (594 AI-generated and 1,185 human-authored), supported by 19,450 review comments and 28,875 review events. We further construct a ten-category taxonomy of rejection reasons grounded in reviewers’ explanations and code-level evidence. Our comparative analysis reveals similarity and differences in failure patterns: AI-generated PRs are more likely to be rejected due to incomplete implementation issues and testing-inadequate issues, human PRs showed a highly similar composition of rejection categories. However, the proportion of the other category is higher for human PRs than for AI PRs. This suggests that rejections of human PRs more often involve heterogeneous or context-specific reasons that do not fall neatly into predefined categories, whereas AI-generated PRs tend to fail in more repetitive and well-defined ways. Failure modes in AI-generated PRs are also more consistent and predictable, with description-level features providing stronger predictive signals compared to human-authored PRs. These findings provide empirical insights for improving AI coding assistants and open new opportunities for designing automated mechanisms and PR-review support tools tailored to AI-generated code.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029 · see citing works
Self-citationAutomatic Generation of Pull Request Descriptions
Liu, Z., Xia, X., Treude, C., Lo, D., & Li, S. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00026 · Cited by 116 · see citing works
Li, H., Zhang, H., & Hassan, A. E. (2025) · arXiv (Cornell University) · doi:10.4230/lipics.esem.2026.67 · Cited by 78 · see citing works
What Do Users Ask in Open-Source AI Repositories? An Empirical Study of GitHub Issues
Yang, Z., Wang, C., Shi, J., Hoang, T., Kochhar, P. S., Lu, Q., Xing, Z., & Lo, D. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00024 · Cited by 36 · see citing works
Self-citationGenerative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions
Xiao, T., Hata, H., Treude, C., & Matsumoto, K. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3643773 · Cited by 29 · see citing works
Pull Request Governance in Open Source Communities
Alami, A., Pardo, R., Cohn, M. L., & Wasowski, A. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3128356 · Cited by 15 · see citing works
"My productivity is boosted, but …" Demystifying Users’ Perception on AI Coding Assistants
Lyu, Y., Yang, Z., Shi, J., Chang, J., Liu, Y., & Lo, D. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00024 · Cited by 9 · see citing works
Self-citationAI-Powered Code Review Assistant for Streamlining Pull Request Merging
Adapa, C., Avulamanda, S. S., Anjana, A., & Victor, A. C. (2024) · IEEE International Conference for Women in Innovation, Technology & Entrepreneurship (ICWITE) · doi:10.1109/icwite59797.2024.10503540 · Cited by 7 · see citing works
Papadakis, N., Patel, A. R., Gottigundala, T., Garro, A., Graham, X., & da Silva, B. (2020) · doi:10.1145/3387940.3392235 · Cited by 5 · see citing works
Qualitatively Analyzing PR Rejection Reasons from Conversations in Open-Source Projects
Gottigundala, T., Sereesathien, S., & da Silva, B. (2021) · IEEE/ACM International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase52884.2021.00021 · Cited by 5 · see citing works
PatchTrack: A comprehensive analysis of ChatGPT’s influence on pull request outcomes
Ogenrwot, D., & Businge, J. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10869-5 · Cited by 1 · see citing works
Li, H., Zhang, H., & Hassan, A. E. (2025) · arXiv · doi:10.48550/arxiv.2507.15003
PatchTrack: A Comprehensive Analysis of ChatGPT's Influence on Pull Request Outcomes
Ogenrwot, D., & Businge, J. (2025) · arXiv · doi:10.48550/arxiv.2505.07700
"My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistants
Lyu, Y., Yang, Z., Shi, J., Chang, J., Liu, Y., & Lo, D. (2025) · arXiv · doi:10.48550/arxiv.2508.12285
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66
Cited by
Still being checked.
- Authors
- Wang, Y., & Yang, Z.
- Affiliations
- University of Alberta (CA)
- ORCID
- Venue
- Proceedings of the 23rd International Conference on Mining Software Repositories
- Publisher
- ACM
- Published
- 2026-04-13
- Pages
- 752-756
- OpenAlex
- W7172011159
- Cited by
- 0
- References
- 12
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- International Conference on Mining Software Repositories (MSR) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- AI-generated code, pull requests, code review, open source software, AI coding assistants
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
