Peer Review on Open-Source Software Projects: Parameters, Statistical Models, and Theory
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Peer review is seen as an important quality-assurance mechanism in both industrial development and the open-source software (OSS) community. The techniques for performing inspections have been well studied in industry; in OSS development, software peer reviews are not as well understood. To develop an empirical understanding of OSS peer review, we examine the review policies of 25 OSS projects and study the archival records of six large, mature, successful OSS projects. We extract a series of measures based on those used in traditional inspection experiments. We measure the frequency of review, the size of the contribution under review, the level of participation during review, the experience and expertise of the individuals involved in the review, the review interval, and the number of issues discussed during review. We create statistical models of the review efficiency, review interval, and effectiveness, the issues discussed during review, to determine which measures have the largest impact on review efficacy. We find that OSS peer reviews are conducted asynchronously by empowered experts who focus on changes that are in their area of expertise. Reviewers provide timely, regular feedback on small changes. The descriptive statistics clearly show that OSS review is drastically different from traditional inspection.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Use of relative code churn measures to predict system defect density
Nagappan, N., & Ball, T. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062514 · Cited by 775 · cited 1×
“In order to determine the relationship between commit activity and review type, we conduct both Pearson and Spearman correlations.3 We assume that commit activity is related to development activity [Nagappan and Ball 2005].”
MethodLee, G. K., & Cole, R. E. (2003) · Organization Science · doi:10.1287/orsc.14.6.633.24866 · Cited by 713
Cited by
An empirical study of the impact of modern code review practices on software quality
McIntosh, S., Kamei, Y., Adams, B., & Hassan, A. E. (2016) · Empirical Software Engineering · doi:10.1007/s10664-015-9381-9 · Cited by 332
Modern code review: a case study at google
Sadowski, C., Söderberg, E., Church, L., Sipko, M., & Bacchelli, A. (2018) · Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice · doi:10.1145/3183519.3183525 · Cited by 280
(2018) · doi:10.1145/3274404 · Cited by 122
“[50] reported that the level of review participation is the most influential factor in the code review efficiency.”
BackgroundReassessing automatic evaluation metrics for code summarization tasks
Roy, D., Fakhoury, S., & Arnaoudova, V. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468588 · Cited by 106
“Useful source code comments play a vital role in program comprehension and other software maintenance activities [45, 58].”
BackgroundCode Review Automation: Strengths and Weaknesses of the State of the Art
Tufano, R., Dabić, O., Mastropaolo, A., Ciniselli, M., & Bavota, G. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3348172 · Cited by 48
Metadata
- Authors
- Rigby, P. C., German, D. M., Cowen, L., & Storey, M.-A.
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2014-09-05
- Volume
- 23
- Issue
- 4
- Pages
- 1-33
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W1982335236
- Cited by
- 110
- References
- 2
- FWCI
- 16.95
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.