The DOI matches its Crossref record.
Open source software peer review practices: a case study of the apache server
Abstract
Peer review is seen as an important quality assurance mechanism in both industrial development and the open source software (OSS) community. The techniques for performing inspections have been well studied in industry; in OSS development, peer reviews are less well understood. We examine the two peer review techniques used by the successful, mature Apache server project: review-then-commit and commit-then-review. Using archival records of email discussion and version control repositories, we construct a series of metrics that produces measures similar to those used in traditional inspection experiments. Specifically, we measure the frequency of review, the level of participation in reviews, the size of the artifact under review, the calendar time to perform a review, and the number of reviews that find defects. We provide a comparison of the two Apache review techniques as well as a comparison of Apache review to inspection in an industrial project. We conclude that Apache reviews can be described as (1) early, frequent reviews (2) of small, independent, complete contributions (3) conducted asynchronously by a potentially large, but actually small, group of self-selected experts (4) leading to an efficient and effective peer review technique.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
The promises and perils of mining GitHub
Kalliamvakou, E., Gousios, G., Blincoe, K., Singer, L., German, D. M., & Damian, D. (2014) · Proceedings of the 11th Working Conference on Mining Software Repositories · doi:10.1145/2597073.2597074 · Cited by 770
Self-citationExpectations, outcomes, and challenges of modern code review
Bacchelli, A., & Bird, C. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606617 · Cited by 650
An exploratory study of the pull-based software development model
Gousios, G., Pinzger, M., & Deursen, A. V. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568260 · Cited by 628
(2013) · doi:10.5555/2486788.2486882 · Cited by 437
McIntosh, S., Kamei, Y., Adams, B., & Hassan, A. E. (2014) · Proceedings of the 11th Working Conference on Mining Software Repositories · doi:10.1145/2597073.2597076 · Cited by 348
Work practices and challenges in pull-based development: the contributor's perspective
Gousios, G., Storey, M.-A., & Bacchelli, A. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884826 · Cited by 335
Self-citationAn empirical study of the impact of modern code review practices on software quality
McIntosh, S., Kamei, Y., Adams, B., & Hassan, A. E. (2016) · Empirical Software Engineering · doi:10.1007/s10664-015-9381-9 · Cited by 332
Work Practices and Challenges in Pull-Based Development: The Integrator's Perspective
Gousios, G., Zaidman, A., Storey, M.-A., & Deursen, A. V. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.55 · Cited by 312
Self-citationThe promises and perils of mining git
Bird, C., Rigby, P. C., Barr, E. T., Hamilton, D. J., German, D. M., & Devanbu, P. (2009) · 2009 6th IEEE International Working Conference on Mining Software Repositories · doi:10.1109/msr.2009.5069475 · Cited by 311
Self-citationConvergent contemporary software peer review practices
Rigby, P. C., & Bird, C. (2013) · Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2491411.2491444 · Cited by 311
Self-citationAn in-depth study of the promises and perils of mining GitHub
Kalliamvakou, E., Gousios, G., Blincoe, K., Singer, L., German, D. M., & Damian, D. (2016) · Empirical Software Engineering · doi:10.1007/s10664-015-9393-5 · Cited by 282
Self-citation
Metadata
- Authors
- Rigby, P. C., German, D. M., & Storey, M.-A.
- Venue
- Proceedings of the 13th international conference on Software engineering - ICSE '08
- Publisher
- ACM Press
- Published
- 2008
- Pages
- 541
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W2133561941
- Cited by
- 237
- References
- 0
- FWCI
- 7.34
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.