The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
On the Use of ChatGPT for Code Review: Do Developers Like Reviews By ChatGPT?
Abstract
Code review is a critical but time-consuming process for ensuring code quality in modern software engineering. To alleviate the effort of reviewing source code, recent studies have investigated the possibility of automating the review process. Moreover, tools based on large language models such as ChatGPT are playing an increasingly important role in this vision. Understanding how these tools are used during code review can provide valuable insights for code review automation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“They found that 30.7% of AI-generated suggestions during code reviews were met with skepticism or disagreement, and most of them were about the generated code.”
From On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub (2026) · cites this paper as Watanabe et al. [41]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Expectations, outcomes, and challenges of modern code review
Bacchelli, A., & Bird, C. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606617 · Cited by 650
McIntosh, S., Kamei, Y., Adams, B., & Hassan, A. E. (2014) · Proceedings of the 11th Working Conference on Mining Software Repositories · doi:10.1145/2597073.2597076 · Cited by 348
An Analysis of the Automatic Bug Fixing Performance of ChatGPT
Sobania, D., Briesch, M., Hanna, C., & Petke, J. (2023) · 2023 IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1109/apr59189.2023.00012 · Cited by 338
An Exploratory Study on Self-Admitted Technical Debt
Potdar, A., & Shihab, E. (2014) · 2014 IEEE International Conference on Software Maintenance and Evolution · doi:10.1109/icsme.2014.31 · Cited by 281
Modern code review: a case study at google
Sadowski, C., Söderberg, E., Church, L., Sipko, M., & Bacchelli, A. (2018) · Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice · doi:10.1145/3183519.3183525 · Cited by 280
Towards Automating Code Review Activities
Tufano, R., Pascarella, L., Tufano, M., Poshyvanyk, D., & Bavota, G. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00027 · Cited by 260
Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., & Peng, X. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623306 · Cited by 235
On Learning Meaningful Code Changes Via Neural Machine Translation
Tufano, M., Pantiuchina, J., Watson, C., Bavota, G., & Poshyvanyk, D. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00021 · Cited by 207
Automating code review activities by large-scale pre-training
Li, Z., Lu, S., Guo, D., Duan, N., Jannu, S., Jenks, G., Majumder, D., Green, J., Svyatkovskiy, A., Fu, S., et al. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549081 · Cited by 196
Code review quality: how developers see it
Kononenko, O., Baysal, O., & Godfrey, M. W. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884840 · Cited by 183
Using pre-trained models to boost code review automation
Tufano, R., Masiero, S., Mastropaolo, A., Pascarella, L., Poshyvanyk, D., & Bavota, G. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510621 · Cited by 177
Identifying the characteristics of vulnerable code changes: an empirical study
Bosu, A., Carver, J. C., Hafiz, M., Hilley, P., & Janni, D. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635880 · Cited by 170
Impact of Peer Code Review on Peer Impression Formation: A Survey
Bosu, A., & Carver, J. C. (2013) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.23 · Cited by 150
Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643674 · Cited by 149
Four eyes are better than two: On the impact of code reviews on software quality
Bavota, G., & Russo, B. (2015) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2015.7332454 · Cited by 147
Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., & Peng, X. (2023) · arXiv · doi:10.48550/arxiv.2309.08221 · Cited by 140
AutoTransform: automated code transformation to support modern code review process
Thongtanunam, P., Pornprasit, C., & Tantithamthavorn, C. K. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510067 · Cited by 108
Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting
Li, T.-O., Zong, W., Wang, Y., Tian, H., Wang, Y., Cheung, S.-C., & Kramer, J. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00089 · Cited by 99
The "Shut the f**k up" Phenomenon: Characterizing Incivility in Open Source Code Review Discussions
Ferreira, I., Cheng, J., & Adams, B. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479497 · Cited by 84
DevGPT: Studying Developer-ChatGPT Conversations
Xiao, T., Treude, C., Hata, H., & Matsumoto, K. (2024) · Proceedings of the 21st International Conference on Mining Software Repositories · doi:10.1145/3643991.3648400 · Cited by 56
Automatic Code Review by Learning the Revision of Source Code
Shi, S., Li, M., Lo, D., Thung, F., & Huo, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014910 · Cited by 54
ToxiSpanSE: An Explainable Toxicity Detection in Code Review Comments
Sarker, J., Sultana, S., Wilson, S. R., & Bosu, A. (2023) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem56168.2023.10304855 · Cited by 45
The Impact of Code Review on Architectural Changes
Paixão, M. H. E., Krinke, J., Han, D., Ragkhitwetsagul, C., & Harman, M. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2912113 · Cited by 44
Towards automating code review at scale
Hellendoorn, V. J., Tsay, J., Mukherjee, M., & Hirzel, M. J. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473134 · Cited by 43
An empirical study on self-admitted technical debt in modern code review
Kashiwa, Y., Nishikawa, R., Kamei, Y., Kondo, M., Shihab, E., Sato, R., & Ubayashi, N. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106855 · Cited by 35
Self-citationCan ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation
Jin, K., Wang, C.-Y., Pham, H. V., & Hemmati, H. (2024) · Proceedings of the 21st International Conference on Mining Software Repositories · doi:10.1145/3643991.3645074 · Cited by 24
An Empirical Investigation on the Performance of Domain Adaptation for T5 Code Completion
Fukumoto, D., Kashiwa, Y., HIRAO, T., Fujiwara, K., & Iida, H. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00073 · Cited by 8
Self-citationLeveraging Execution Trace with ChatGPT: A Case Study on Automated Fault Diagnosis
Sakura, T., Soga, R., Kanuka, H., Shimari, K., & Ishio, T. (2023) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58846.2023.00052 · Cited by 3
Research works printed but not linked: 5
- 10.5555/3155562.3155579 · doi:10.5555/3155562.3155579
- Eman Abdullah AlOmar, Anushkrishna Venkatakrishnan, Mohamed Wiem Mkaouer, Christian D. Newman, and Ali Ouni. 2024. How to Refactor this Code? An Exploratory Study on Developer-ChatGPT Refactoring Conversations. CoRR abs/2402.06013 (2024). · arXiv
- Carolyn D. Egelman, Emerson R. Murphy-Hill, Elizabeth Kammer, Margaret Morrow Hodges, Collin Green, Ciera Jaspan, and James Lin. 2020. Predicting developers’ negative feelings about code review. In Proc. of the 42nd International Conference on Software Engineering (ICSE’20). 174–185.
- Samia Kabir, David N. Udo-Imeh, Bonan Kou, and Tianyi Zhang. 2023. Who Answers It Better? An In-Depth Analysis of ChatGPT and Stack Overflow Answers to Software Engineering Questions. CoRR abs/2308.02312 (2023). · arXiv
- Peter C. Rigby and Christian Bird. 2013. Convergent contemporary software peer review practices. In Proc. of the 2013 Joint Meeting of the European Software Engineering Conference (FSE’13). 202–212.
Cited by
On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
Watanabe, M., Li, H., Kashiwa, Y., Reid, B., Iida, H., & Hassan, A. E. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3798166 · Cited by 34
Self-citation
Metadata
- Authors
- Watanabe, M., Kashiwa, Y., Lin, B., Hirao, T., Yamaguchi, K., & Iida, H.
- Affiliations
- Nara Institute of Science and Technology, Japan
- Radboud University, Netherlands
- Nara College, National Institute of Technology, Japan
- Venue
- Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2024-06-18
- Pages
- 375-380
- Cited by
- 28
- References
- 29
- Funders
- Japan Science and Technology Agency (JPMJPR22P3)
- Japan Society for the Promotion of Science (P21H03416, JP24K02921, JPJSBP120239929)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.