CoChat
Sign Up
PreprintCited by 0Software Engineering ResearchSoftware Engineering Techniques and PracticesSoftware Testing and Debugging Techniques

SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback

Open access
Kumar, Deepak·2026·arXiv
DOI 10.48550/arxiv.2603.26130arXiv 2603.26130
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce SWE-PRBench, a benchmark of 350 pull requests with human-annotated ground truth for evaluating AI code review quality. Evaluated against an LLM-as-judge framework validated at kappa=0.75, 8 frontier models detect only 15-31% of human-flagged issues on the diff-only configuration, demonstrating that AI code review remains far below human expert performance despite strong results on code generation benchmarks. Pull requests are drawn from active open-source repositories, filtered from 700 candidates using a Repository Quality Score, and evaluated under three frozen context configurations: diff only (config_A), diff with file content (config_B), and full context (config_C), enabling systematic ablation of context provision strategies. All 8 models degrade monotonically from config_A to config_C, even when context is provided via structured semantic layers including AST-extracted function context and import graph resolution. The dominant mechanism is a collapse of Type2_Contextual issue detection at config_B, consistent with attention dilution in long contexts: a structured 2,000-token diff-with-summary prompt outperforms a 2,500-token full-context prompt enriched with execution context, behaviour mapping, and test signatures across all 8 models. The top four models are statistically indistinguishable (mean score 0.147-0.153) while a clear tier gap separates them from the remaining four (mean score <= 0.113). Dataset, contexts, annotations, and evaluation harness are released publicly.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2603.26130

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Research works printed but not linked: 16

  1. A. Bacchelli and C. Bird, "Expectations, outcomes, and challenges of modern code review," in Proc. ICSE, 2013, pp. 712-721.
  2. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, "SWE-bench: Can language models resolve real-world GitHub issues?" in Proc. ICLR, 2024.
  3. M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, "Using pre-trained models to boost code review automation," in Proc. ICSE, 2022, pp. 1-12.
  4. Z. Li, S. Lu, S. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu, and N. Sundaresan, "Automating code review activities by large-scale pre-training," in Proc. ESEC/FSE, 2022, pp. 1035-1047.
  5. R. Tufano, S. Gualandi, M. Aghajani, M. Monperrus, and A. Bacchelli, "Code review au- tomation: Strengths and weaknesses of the state of the art," IEEE Transactions on Software Engineering, vol. 50, no. 5, 2024.
  6. O. Kononenko, O. Baysal, and M. W. Godfrey, "Code review quality: How developers see it," in Proc. ICSE, 2016, pp. 1028-1038.
  7. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, "Judging LLM-as-a-judge with MT-Bench and chatbot arena," in Proc. NeurIPS, 2023.
  8. Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, "G-Eval: NLG evaluation using GPT-4 with better human alignment," in Proc. EMNLP, 2023, pp. 2511-2522.
  9. J. Cohen, "A coefficient of agreement for nominal scales," Educational and Psychological Measurement, vol. 20, no. 1, pp. 37-46, 1960.
  10. J. Lu, D. Li, B. Yu, and L. Zhang, "DeepCRCEval: Revisiting the evaluation of code review comment generation," arXiv preprint arXiv:2412.18291, 2025. · arXiv
  11. M. Golzadeh, A. Decan, D. Legay, and T. Mens, "A ground-truth dataset and classification model for detecting bots in GitHub issue and PR comments," Journal of Systems and Software, vol. 175, 2021.
  12. A. Bosu, M. Greiler, and C. Bird, "Characteristics of useful code reviews: An empirical study at Microsoft," in Proc. MSR, 2015, pp. 146-156.
  13. Y. Zhang, Y. Zhang, Z. Sun, Y. Jiang, and H. Liu, "LAURA: Enhancing code review generation with context-enriched retrieval-augmented LLM," arXiv preprint arXiv:2512.01356, 2024. · arXiv
  14. R. Tan, J. Fu, T. Pang, L. Du, M. Lin, and C. Gao, "JudgeBench: A benchmark for evaluating LLM-based judges," arXiv preprint arXiv:2410.12784, 2024. · arXiv
  15. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, "Lost in the middle: How language models use long contexts," Transactions of the Association for Computational Linguistics, vol. 12, pp. 157-173, 2024.
  16. OpenAI, "OpenAI API pricing," 2024. [Online]. Available: https://openai.com/api/ pricing/ A PR Record Schema Each SWE-PRBench pull request is stored as a JSON record with five field categories. Meta- data: task id, repo, pr number, language, pr type, difficulty, merged at. Scoring sig- nals: rvs score and five component scores (review depth, code complexity, discussion signal, test change signal, bug fix signal), plus lines added, lines removed, files changed. Re- producibility anchors: base commit and head commit SHA hashes pinning the exact file state at review time. Ground truth: human review comments array with per-comment fields for author, body, path, line, diff hunk, and reply status; plus num substantive comments, num unique reviewers, has requested changes, ai comments removed. Diff: diff patch as the raw unified diff retrieved from the GitHub REST API; agent input is populated by the context builder at evaluation time.

Authors
Kumar, D.
Corresponding author
Kumar, Deepak
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2603.26130
arXiv
2603.26130
OpenAlex
W7143378675
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
pull requests, benchmark, large language models, code quality evaluation, context length, attention dilution, LLM-as-a-judge, open source software

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review