The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings
Abstract
Software defect prediction strives to improve software quality and testing efficiency by constructing predictive classification models from code attributes to enable a timely identification of fault-prone modules. Several classification models have been evaluated for this task. However, due to inconsistent findings regarding the superiority of one classifier over another and the usefulness of metric-based classification in general, more research is needed to improve convergence across studies and further advance confidence in experimental results. We consider three potential sources for bias: comparing classifiers over one or a small number of proprietary data sets, relying on accuracy indicators that are conceptually inappropriate for software defect prediction and cross-study comparisons, and, finally, limited use of statistical testing procedures to secure empirical findings. To remedy these problems, a framework for comparative software defect prediction experiments is proposed and applied in a large-scale empirical comparison of 22 classifiers over 10 public domain data sets from the NASA Metrics Data repository. Overall, an appealing degree of predictive accuracy is observed, which supports the view that metric-based classification is useful. However, our results indicate that the importance of the particular classification algorithm may be less than previously assumed since no significant performance differences could be detected among the top 17 classifiers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
Neural networks for pattern recognition
(1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,538
Feinleib, M., & Zar, J. H. (1975) · Journal of the American Statistical Association · doi:10.2307/2285423 · Cited by 12,680
The Elements of Statistical Learning: Data Mining, Inference, and Prediction
John Lu, Z. Q. (2010) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2010.00646_6.x · Cited by 12,390
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
Least Squares Support Vector Machine Classifiers
Suykens, J. A. K., & Vandewalle, J. (1999) · Neural Processing Letters · doi:10.1023/a:1018628609742 · Cited by 9,362
The use of the area under the ROC curve in the evaluation of machine learning algorithms
Bradley, A. P. (1997) · Pattern Recognition · doi:10.1016/s0031-3203(96)00142-2 · Cited by 7,217
McCabe, T. J. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233837 · Cited by 6,059
Cited by
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154
An experimental comparison of classification algorithms for imbalanced credit scoring data sets
Brown, I., & Mues, C. (2012) · Expert Systems with Applications · doi:10.1016/j.eswa.2011.09.033 · Cited by 730
Self-citationOn the relative value of cross-company and within-company data for defect prediction
Turhan, B., Menzies, T., Bener, A. B., & Di Stefano, J. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9103-7 · Cited by 725
A large-scale empirical study of just-in-time quality assurance
Kamei, Y., Shihab, E., Adams, B., Hassan, A. E., Mockus, A., Sinha, A., & Ubayashi, N. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.70 · Cited by 715
An exploratory study of the pull-based software development model
Gousios, G., Pinzger, M., & Deursen, A. V. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568260 · Cited by 628
Evaluating defect prediction approaches: a benchmark and an extensive comparison
D’Ambros, M., Lanza, M., & Robbes, R. (2012) · Empirical Software Engineering · doi:10.1007/s10664-011-9173-9 · Cited by 626
A systematic review of machine learning techniques for software fault prediction
Malhotra, R. (2015) · Applied Soft Computing · doi:10.1016/j.asoc.2014.11.023 · Cited by 620
Shin, Y., Meneely, A., Williams, L., & Osborne, J. A. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.81 · Cited by 607
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2584050 · Cited by 574
Data Quality: Some Comments on the NASA Software Defect Datasets
Shepperd, M., Song, Q., Sun, Z., & Mair, C. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.11 · Cited by 570
Software fault prediction metrics: A systematic literature review
Radjenović, D., Heričko, M., Torkar, R., & Živkovič, A. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.02.009 · Cited by 538
Defect prediction from static code features: current results, limitations, new approaches
Menzies, T., Milton, Z., Turhan, B., Cukic, B., Jiang, Y., & Bener, A. (2010) · Automated Software Engineering · doi:10.1007/s10515-010-0069-5 · Cited by 466
Verbeke, W., Dejaeger, K., Martens, D., Hur, J., & Baesens, B. (2012) · European Journal of Operational Research · doi:10.1016/j.ejor.2011.09.031 · Cited by 462
Self-citationComparing and experimenting machine learning techniques for code smell detection
Arcelli Fontana, F., Mäntylä, M. V., Zanoni, M., & Marino, A. (2016) · Empirical Software Engineering · doi:10.1007/s10664-015-9378-4 · Cited by 450
The Impact of Automated Parameter Optimization on Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2794977 · Cited by 415
Software defect prediction using ensemble learning on selected features
Laradji, I. H., Alshayeb, M., & Ghouti, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2014.07.005 · Cited by 380
A General Software Defect-Proneness Prediction Framework
Song, Q., Jia, Z., Shepperd, M., Ying, S., & Liu, J. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.90 · Cited by 374
Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.91 · Cited by 364
Researcher Bias: The Use of Machine Learning in Software Defect Prediction
Shepperd, M., Bowes, D., & Hall, T. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2322358 · Cited by 359
An investigation on the feasibility of cross-project defect prediction
He, Z., Shu, F., Yang, Y., Li, M., & Wang, Q. (2012) · Automated Software Engineering · doi:10.1007/s10515-011-0090-3 · Cited by 333
Automated parameter optimization of classification techniques for defect prediction models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884857 · Cited by 331
Software defect prediction using Bayesian networks
Okutan, A., & Yıldız, O. T. (2014) · Empirical Software Engineering · doi:10.1007/s10664-012-9218-8 · Cited by 330
Tantithamthavorn, C., Hassan, A. E., & Matsumoto, K. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876537 · Cited by 327
A Systematic Literature Review and Meta-Analysis on Cross Project Defect Prediction
Hosseini, S., Turhan, B., & Gunarathna, D. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2770124 · Cited by 317
An empirical study on software defect prediction with a simplified metric set
He, P., Li, B., Liu, X., Chen, J., & Ma, Y. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2014.11.006 · Cited by 309
Bennin, K. E., Keung, J., Phannachitta, P., Monden, A., & Mensah, S. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2731766 · Cited by 298
Choosing software metrics for defect prediction: an investigation on feature selection techniques
Gao, K., Khoshgoftaar, T. M., Wang, H., & Seliya, N. (2011) · Software: Practice and Experience · doi:10.1002/spe.1043 · Cited by 281
Heterogeneous Defect Prediction
Nam, J., Fu, W., Kim, S., Menzies, T., & Tan, L. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2720603 · Cited by 277
Explaining Data-Driven Document Classifications1
Martens, D., & Provost, F. (2014) · MIS Quarterly · doi:10.25300/misq/2014/38.1.04 · Cited by 264
“…studies have revealed that when it comes to predictive accuracy, non-linear methods often outperform traditional statistical methods such as multiple regression, logistic regression, naive Bayesian and linear discriminant analysis (see, e.g. Baesens et al. (2003), Lessmann et al. (2008)).”
MethodProgress on approaches to software defect prediction
Li, Z., Jing, X.-Y., & Zhu, X. (2018) · IET Software · doi:10.1049/iet-sen.2017.0148 · Cited by 264
Data mining techniques for software effort estimation: a comparative study
Dejaeger, K., Verbeke, W., Martens, D., & Baesens, B. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.55 · Cited by 224
Self-citationA simpler model of software readability
Posnett, D., Hindle, A., & Devanbu, P. (2011) · Proceedings of the 8th Working Conference on Mining Software Repositories · doi:10.1145/1985441.1985454 · Cited by 182
An Empirical Study of Model-Agnostic Techniques for Defect Prediction Models
Jiarpakdee, J., Tantithamthavorn, C. K., Dam, H. K., & Grundy, J. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2982385 · Cited by 148
Wahono (2007) · doi:10.3923/jse.2007.1.12 · Cited by 113
A reference model for customer-centric data mining with support vector machines
Lessmann, S., & Voß, S. (2009) · European Journal of Operational Research · doi:10.1016/j.ejor.2008.12.017 · Cited by 86
Self-citationDesign Rule Spaces: A New Model for Representing and Analyzing Software Architecture
Cai, Y., Xiao, L., Kazman, R., Mo, R., & Feng, Q. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2797899 · Cited by 39
“[33] used 39 complexity-based metrics and applied 22 classifiers to improve the defect prediction.”
MethodWhat are the characteristics of highly-selected packages? A case study on the npm ecosystem
Mujahid, S., Abdalkareem, R., & Shihab, E. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111588 · Cited by 16
Metadata
- Authors
- Lessmann, S., Baesens, B., Mues, C., & Pietsch, S.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2008-07
- Volume
- 34
- Issue
- 4
- Pages
- 485-496
- ISSN
- 0098-5589 (ISSN-L)
- OpenAlex
- W2105776892
- Cited by
- 1,226
- References
- 66
- FWCI
- 123.30
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.