The DOI matches its Crossref record; 89 of the 89 references it cites were found in a registry or an index.
Researcher Bias: The Use of Machine Learning in Software Defect Prediction
Open accessAbstract
Background. The ability to predict defect-prone software components would be valuable. Consequently, there have been many empirical studies to evaluate the performance of different techniques endeavouring to accomplish this effectively. However no one technique dominates and so designing a reliable defect prediction model remains problematic. Objective. We seek to make sense of the many conflicting experimental results and understand which factors have the largest effect on predictive performance. Method. We conduct a meta-analysis of all relevant, high quality primary studies of defect prediction to determine what factors influence predictive performance. This is based on 42 primary studies that satisfy our inclusion criteria that collectively report 600 sets of empirical prediction results. By reverse engineering a common response variable we build a random effects ANOVA model to examine the relative contribution of four model building factors (classifier, data set, input metrics and researcher group) to model prediction performance. Results. Surprisingly we find that the choice of classifier has little impact upon performance (1.3 percent) and in contrast the major (31 percent) explanatory factor is the researcher group. It matters more who does the work than what is done. Conclusion. To overcome this high level of researcher bias, defect prediction researchers should (i) conduct blind analysis, (ii) improve reporting protocols and (iii) conduct more intergroup studies in order to alleviate expertise issues. Lastly, research is required to determine whether this bias is prevalent in other applications domains.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,245
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
A metrics suite for object oriented design
Chidamber, S. R., & Kemerer, C. F. (1994) · IEEE Transactions on Software Engineering · doi:10.1109/32.295895 · Cited by 5,669
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Powers, D. M. W. (2020) · arXiv · doi:10.48550/arxiv.2010.16061 · Cited by 5,440
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Assessing the accuracy of prediction algorithms for classification: an overview
Baldi, P., Brunak, S., Chauvin, Y., Andersen, C. A. F., & Nielsen, H. (2000) · Bioinformatics · doi:10.1093/bioinformatics/16.5.412 · Cited by 2,111
Sensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation
Rodriguez, J. D., Perez, A., & Lozano, J. A. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.187 · Cited by 1,890 · cited 1×
“dency (see [10] for a more detailed discussion).”
BackgroundNonparametric Statistics for the Behavioral Sciences
(2010) · Encyclopedia of Research Design · doi:10.4135/9781412961288.n273 · Cited by 1,414
Cited by
The Impact of Automated Parameter Optimization on Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2794977 · Cited by 415
Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.91 · Cited by 364
Automated parameter optimization of classification techniques for defect prediction models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884857 · Cited by 331
A Comprehensive Investigation of the Role of Imbalanced Learning for Software Defect Prediction
Song, Q., Guo, Y., & Shepperd, M. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2836442 · Cited by 295
Self-citationProgress on approaches to software defect prediction
Li, Z., Jing, X.-Y., & Zhu, X. (2018) · IET Software · doi:10.1049/iet-sen.2017.0148 · Cited by 264
Metadata
- Authors
- Shepperd, M., Bowes, D., & Hall, T.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2014-06-01
- Volume
- 40
- Issue
- 6
- Pages
- 603-616
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2118283821
- Cited by
- 359
- References
- 89
- FWCI
- 71.19
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Researcher Bias: The Use of Machine Learning in Software Defect Prediction. https://app.cochat.ai/papers/10.1109/tse.2014.2322358
- The paper
- Shepperd, M., Bowes, D., & Hall, T. (2014). Researcher Bias: The Use of Machine Learning in Software Defect Prediction. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2014.2322358
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.