Expectations, outcomes, and challenges of modern code review
Open accessAbstract
Code review is a common software engineering practice employed both in open source and industrial contexts. Review today is less formal and more “lightweight” than the code inspections performed and studied in the 70s and 80s. We empirically explore the motivations, challenges, and outcomes of tool-based code reviews. We observed, interviewed, and surveyed developers and managers and manually classified hundreds of review comments across diverse teams at Microsoft. Our study reveals that while finding defects remains the main motivation for review, reviews are less about defects than expected and instead provide additional benefits such as knowledge transfer, increased team awareness, and creation of alternative solutions to problems. Moreover, we find that code and change understanding is the key aspect of code reviewing and that developers employ a wide range of mechanisms to meet their understanding needs, most of which are not met by current tools. We provide recommendations for practitioners and researchers.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Code comprehension and understandability are a central challenge in effective code review.
“In addition, code comprehension and understandability have been found to be a central challenge in effective code review [5]”
From Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code (2026) · cites this paper as [5]
Cites
The Hawthorne effect: A reconsideration of the methodological artifact.
Adair, J. G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.2.334 · Cited by 1,095 · cited 1× · see citing works
“used to analyze real code review comments without incurring the Hawthorne effect [3].”
MethodUsing grounded theory to study the experience of software development
Adolph, S., Hall, W., & Kruchten, P. (2011) · Empirical Software Engineering · doi:10.1007/s10664-010-9152-6 · Cited by 235 · cited 1× · see citing works
“(also used in grounded theory [22]): breaking up the answers into smaller coherent units (sentences or paragraphs) and adding codes to them.”
Flagged first, then by citations
Cited by
Sorted by citations
Züger, M., Müller, S. C., Meyer, A. N., & Fritz, T. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174165 · Cited by 66 · see citing works
An Actionable Framework for Understanding and Improving Developer Experience
Greiler, M., Storey, M.-A., & Noda, A. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3175660 · Cited by 57 · see citing works
How Do Users Revise Answers on Technical Q&A Websites? A Case Study on Stack Overflow
Wang, S., Chen, T.-H., & Hassan, A. E. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2874470 · Cited by 50 · see citing works
Understanding Accessibility and Collaboration in Programming for People with Visual Impairments
Pandey, M., Kameswaran, V., Rao, H. V., O'Modhrain, S., & Oney, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449203 · Cited by 54 · see citing works
Code Review Automation: Strengths and Weaknesses of the State of the Art
Tufano, R., Dabić, O., Mastropaolo, A., Ciniselli, M., & Bavota, G. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3348172 · Cited by 48 · see citing works
The pushback effects of race, ethnicity, gender, and age in code review
Murphy-Hill, E., Jaspan, C., Egelman, C., & Cheng, L. (2022) · Communications of the ACM · doi:10.1145/3474097 · Cited by 30 · see citing works
Why Are AI Agent–Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study
Alam, K., Mondal, S., & Roy, B. (2026) · Proceedings of the 23rd International Conference on Mining Software Repositories · doi:10.1145/3793302.3793611
Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code
Khojah, R., Gomes de Oliveira Neto, F., Mohamad, M., Frattini, J., & Leitner, P. (2026) · Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3832783.3834423
Who Pays the Review Cost? Triage, Fairness, and Accountability in AI-authored Pull Requests
Rahman, M. S., Alam, K., Roy, B., & Roy, C. K. (2026) · arXiv · doi:10.48550/arxiv.2610.11179
Plausible Is Not Enough: Trust and Evidence in AI-Authored Pull Request Review
Rahman, M. S., Alam, K., Roy, B., & Roy, C. K. (2026) · arXiv · doi:10.48550/arxiv.2610.21144
- Authors
- Bacchelli, A., & Bird, C.
- Affiliations
- Università della Svizzera italiana (CH)
- Microsoft (United States) (US)
- Venue
- 2013 35th International Conference on Software Engineering (ICSE)
- Publisher
- IEEE
- Published
- 2013-05
- Pages
- 712-721
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4235786747
- Cited by
- 650
- References
- 2
- Citations by year
- 2026: 75 · 2025: 57 · 2024: 76 · 2023: 66 · 2022: 63 · 2021: 64 · 2020: 52 · 2019: 51 · 2018: 29 · 2017: 39 · 2016: 28 · 2015: 41 · 2014: 7 · 2013: 2
- FWCI
- 20.64
- Type
- conference paper
- Language
- en
- Open access
- green
- www.zora.uzh.ch
- Licence
- other-oa
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, other-oa)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, other-oa)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, other-oa)
- Zurich Open Repository and Archive (University of Zurich)
- Zenodo (CERN European Organization for Nuclear Research) (other-oa)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- code review, knowledge transfer, team awareness, reviewer comments, developer practices, software engineering practices
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
