The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects
Abstract
Software code review, i.e., the practice of having third-party team members critique changes to a software system, is a well-established best practice in both open source and proprietary software domains. Prior work has shown that the formal code inspections of the past tend to improve the quality of software delivered by students and small teams. However, the formal code inspection process mandates strict review criteria (e.g., in-person meetings and reviewer checklists) to ensure a base level of review quality, while the modern, lightweight code reviewing process does not. Although recent work explores the modern code review process qualitatively, little research quantitatively explores the relationship between properties of the modern code review process and software quality. Hence, in this paper, we study the relationship between software quality and: (1) code review coverage, i.e., the proportion of changes that have been code reviewed, and (2) code review participation, i.e., the degree of reviewer involvement in the code review process. Through a case study of the Qt, VTK, and ITK projects, we find that both code review coverage and participation share a significant link with software quality. Low code review coverage and participation are estimated to produce components with up to two and five additional post-release defects respectively. Our results empirically confirm the intuition that poorly reviewed code has a negative impact on software quality in large systems using modern reviewing tools.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
McCabe, T. J. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233837 · Cited by 6,059
ANew Look at the Statistical Model Identification
AKAlKE, H. (2011) · Cited by 4,177
Applied Regression Analysis and Generalized Linear Models
Fox, J. (2008) · Cited by 2,283
Chambers, J., Hastie, T., & Pregibon, D. (1990) · Compstat · doi:10.1007/978-3-642-50096-1_48 · Cited by 1,742
Two case studies of open source software development: Apache and Mozilla
Mockus, A., Fielding, R. T., & Herbsleb, J. D. (2002) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/567793.567795 · Cited by 1,685
Design and code inspections to reduce errors in program development
Fagan, M. E. (1976) · IBM Systems Journal · doi:10.1147/sj.153.0182 · Cited by 1,492
An R companion to applied regression
Fox, J., & Weisberg, S. (2011) · Cited by 1,349
Use of relative code churn measures to predict system defect density
Nagappan, N., & Ball, T. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062514 · Cited by 775 · cited 1×
“Components that have undergone a lot of change are likely defectprone [29, 30].”
BackgroundPredicting fault incidence using software change history
Graves, T. L., Karr, A. F., Marron, J. S., & Siy, H. (2000) · IEEE Transactions on Software Engineering · doi:10.1109/32.859533 · Cited by 753
A large-scale empirical study of just-in-time quality assurance
Kamei, Y., Shihab, E., Adams, B., Hassan, A. E., Mockus, A., Sinha, A., & Ubayashi, N. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.70 · Cited by 715
Self-citationPredicting faults using the complexity of code changes
Hassan, A. E. (2009) · 2009 IEEE 31st International Conference on Software Engineering · doi:10.1109/icse.2009.5070510 · Cited by 708
Self-citationClassifying Software Changes: Clean or Buggy?
Kim, S., Whitehead, & Zhang, Y. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.70773 · Cited by 672
Design and Code Inspections to Reduce Errors in Program Development
Fagan, M. E. (2001) · Pioneers and Their Contributions to Software Engineering · doi:10.1007/978-3-642-48354-7_13 · Cited by 561
(2013) · doi:10.5555/2486788.2486882 · Cited by 437
Predicting risk of software changes
Mockus, A., & Weiss, D. M. (2002) · Bell Labs Technical Journal · doi:10.1002/bltj.2229 · Cited by 429
Don't touch my code!: examining the effects of ownership on software quality
Bird, C., Nagappan, N., Murphy, B., Gall, H., & Devanbu, P. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025119 · Cited by 340
Convergent contemporary software peer review practices
Rigby, P. C., & Bird, C. (2013) · Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2491411.2491444 · Cited by 311
Software Dependencies, Work Dependencies, and Their Impact on Failures
Cataldo, M., Mockus, A., Roberts, J. A., & Herbsleb, J. D. (2009) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2009.42 · Cited by 271
Open source software peer review practices: a case study of the apache server
Rigby, P. C., German, D. M., & Storey, M.-A. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368162 · Cited by 237
Revisiting common bug prediction findings using effort-aware models
Kamei, Y., Matsumoto, S., Monden, A., Matsumoto, K.-I., Adams, B., & Hassan, A. E. (2010) · 2010 IEEE International Conference on Software Maintenance · doi:10.1109/icsm.2010.5609530 · Cited by 207
Self-citationOwnership, experience and defects: a fine-grained study of authorship
Rahman, F., & Devanbu, P. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985860 · Cited by 203
Robust Regression in R An Appendix to An R Companion to Applied Regression, Second Edition
Fox, J., & Weisberg, S. (2011) · Cited by 200
Understanding broadcast based peer review on open source software projects
Rigby, P. C., & Storey, M.-A. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985867 · Cited by 194
How, and Why, Process Metrics are better
Rahman, F., & Dévanbu, P. (2013) · Cited by 184
An Investigation into the Functional Form of the Size-Defect Relationship for Software Modules
Koru, A. G., Zhang, D., Emam, K. E., & Liu, H. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.90 · Cited by 174
(2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.87 · Cited by 170
The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data
Kemerer, C. F., & Paulk, M. C. (2009) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2009.27 · Cited by 167
Organizational volatility and its effects on software defects
Mockus, A. (2010) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1882291.1882311 · Cited by 144
Will my patch make it? And how fast? Case study on the Linux kernel
Jiang, Y., Adams, B., & Germán, D. M. (2013) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2013.6624016 · Cited by 140
Self-citationAn analysis of developer metrics for fault prediction
Matsumoto, S., Kamei, Y., Monden, A., Matsumoto, K., & Nakamura, M. (2010) · doi:10.1145/1868328.1868356 · Cited by 115
Self-citationShihab, E., Jiang, Z. M., Ibrahim, W. M., Adams, B., & Hassan, A. E. (2010) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1852786.1852792 · Cited by 111
Self-citationThe influence of non-technical factors on code review
Baysal, O., Kononenko, O., Holmes, R., & Godfrey, M. W. (2013) · Working Conference on Reverse Engineering (WCRE) · doi:10.1109/wcre.2013.6671287 · Cited by 96
High-impact defects: a study of breakage and surprise defects
Shihab, E., Mockus, A., Kamei, Y., Adams, B., & Hassan, A. E. (2011) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2025113.2025155 · Cited by 96
Self-citationTowards a simplification of the bug report form in eclipse
Herraiz, I., Germán, D. M., Gonzalez-Barahona, J. M., & Robles, G. (2008) · doi:10.1145/1370750.1370786 · Cited by 66
Menzies, T., Stefano, J. S. D., Chapman, M., & McGill, K. (2004) · IEEE Software Engineering Workshop (SEW) · doi:10.1109/sew.2002.1199449 · Cited by 62
Management of community contributions: A case study on the Android and Linux software ecosystems
Bettenburg, N., Hassan, A. E., Adams, B., & Germán, D. M. (2013) · Empirical Software Engineering · doi:10.1007/s10664-013-9284-6 · Cited by 38
Self-citation
Cited by
fMRIPrep: A robust preprocessing pipeline for functional MRI
Esteban, O., Markiewicz, C. J., Blair, R. W., Moodie, C. A., Isik, A. I., Erramuzpe, A., Kent, J. D., Goncalves, M., DuPre, E., Snyder, M., et al. (2019) · Nature Methods · doi:10.1038/s41592-018-0235-4 · Cited by 3,905
The Impact of Automated Parameter Optimization on Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2794977 · Cited by 415
Self-citationAn empirical study of the impact of modern code review practices on software quality
McIntosh, S., Kamei, Y., Adams, B., & Hassan, A. E. (2016) · Empirical Software Engineering · doi:10.1007/s10664-015-9381-9 · Cited by 332
Self-citationTantithamthavorn, C., Hassan, A. E., & Matsumoto, K. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876537 · Cited by 327
Self-citationAn Empirical Study of Model-Agnostic Techniques for Defect Prediction Models
Jiarpakdee, J., Tantithamthavorn, C. K., Dam, H. K., & Grundy, J. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2982385 · Cited by 148
Metadata
- Authors
- McIntosh, S., Kamei, Y., Adams, B., & Hassan, A. E.
- Venue
- Proceedings of the 11th Working Conference on Mining Software Repositories
- Publisher
- ACM
- Published
- 2014-05-31
- Pages
- 192-201
- OpenAlex
- W2108769867
- Cited by
- 348
- References
- 39
- FWCI
- 29.64
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.