The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings
Abstract
Software defect prediction strives to improve software quality and testing efficiency by constructing predictive classification models from code attributes to enable a timely identification of fault-prone modules. Several classification models have been evaluated for this task. However, due to inconsistent findings regarding the superiority of one classifier over another and the usefulness of metric-based classification in general, more research is needed to improve convergence across studies and further advance confidence in experimental results. We consider three potential sources for bias: comparing classifiers over one or a small number of proprietary data sets, relying on accuracy indicators that are conceptually inappropriate for software defect prediction and cross-study comparisons, and, finally, limited use of statistical testing procedures to secure empirical findings. To remedy these problems, a framework for comparative software defect prediction experiments is proposed and applied in a large-scale empirical comparison of 22 classifiers over 10 public domain data sets from the NASA Metrics Data repository. Overall, an appealing degree of predictive accuracy is observed, which supports the view that metric-based classification is useful. However, our results indicate that the importance of the particular classification algorithm may be less than previously assumed since no significant performance differences could be detected among the top 17 classifiers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
Neural networks for pattern recognition
(1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,538
Feinleib, M., & Zar, J. H. (1975) · Journal of the American Statistical Association · doi:10.2307/2285423 · Cited by 12,680
The Elements of Statistical Learning: Data Mining, Inference, and Prediction
John Lu, Z. Q. (2010) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2010.00646_6.x · Cited by 12,390
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
Least Squares Support Vector Machine Classifiers
Suykens, J. A. K., & Vandewalle, J. (1999) · Neural Processing Letters · doi:10.1023/a:1018628609742 · Cited by 9,362
The use of the area under the ROC curve in the evaluation of machine learning algorithms
Bradley, A. P. (1997) · Pattern Recognition · doi:10.1016/s0031-3203(96)00142-2 · Cited by 7,217
McCabe, T. J. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233837 · Cited by 6,059
(1996) · Cited by 4,738
(2024) · Physics Subject Headings (PhySH) · doi:10.29172/7c2a6982-6d72-4cd8-bba6-2fccb06a7011 · Cited by 3,857
Abe, S. (2001) · Springer eBooks · doi:10.1007/978-1-4471-0285-4 · Cited by 3,721
Advances in neural information processing systems 11
Peter Sollich, A. (1999) · Cited by 3,168
Halstead, M. H. (1977) · CERN Document Server (European Organization for Nuclear Research) · Cited by 2,339
Supervised and Unsupervised Discretization of Continuous Features
Dougherty, J., Kohavi, R., & Sahami, M. (1995) · Machine Learning Proceedings 1995 · doi:10.1016/b978-1-55860-377-6.50032-3 · Cited by 1,936
A validation of object-oriented design metrics as quality indicators
Basili, V. R., Briand, L. C., & Melo, W. L. (1996) · IEEE Transactions on Software Engineering · doi:10.1109/32.544352 · Cited by 1,726
Robust classification for imprecise environments
Provost, F., & Fawcett, T. (2001) · Machine Learning · doi:10.1023/a:1007601015854 · Cited by 1,250
Benchmarking attribute selection techniques for discrete class data mining
Hall, M. A., & Holmes, G. (2003) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2003.1245283 · Cited by 1,232
Data Mining Static Code Attributes to Learn Defect Predictors
Menzies, T., Greenwald, J., & Frank, A. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.256941 · Cited by 1,215
Landwehr, N., Hall, M., & Frank, E. (2005) · Machine Learning · doi:10.1007/s10994-005-0466-3 · Cited by 1,214
Statistical Comparisons of Classifiers over Multiple Data Sets
DemšarJanez (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248548 · Cited by 1,074
Large margin classification using the perceptron algorithm
Freund, Y., & Schapire, R. E. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279985 · Cited by 1,071
A critique of software defect prediction models
Fenton, N. E., & Neil, M. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.815326 · Cited by 1,040
YALE: rapid prototyping for complex data mining tasks
Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150531 · Cited by 1,010
Estimating software project effort using analogies
Shepperd, M., & Schofield, C. (1997) · IEEE Transactions on Software Engineering · doi:10.1109/32.637387 · Cited by 954
Tipping, M. E. (1999) · Cited by 912
Benchmarking state-of-the-art classification algorithms for credit scoring
Baesens, B., Van Gestel, T., Viaene, S., Stepanova, M., Suykens, J., & Vanthienen, J. (2003) · Journal of the Operational Research Society · doi:10.1057/palgrave.jors.2601545 · Cited by 904
Self-citationThe Alternating Decision Tree Learning Algorithm
Freund, Y., & Mason, L. (1999) · Cited by 773
K*: An Instance-based Learner Using an Entropic Distance Measure
Cleary, J. G., & Trigg, L. E. (1995) · Machine Learning Proceedings 1995 · doi:10.1016/b978-1-55860-377-6.50022-0 · Cited by 753
Benchmarking Least Squares Support Vector Machine Classifiers
van Gestel, T., Suykens, J. A. K., Baesens, B., Viaene, S., Vanthienen, J., Dedene, G., de Moor, B., & Vandewalle, J. (2004) · Machine Learning · doi:10.1023/b:mach.0000008082.80494.e0 · Cited by 745
Self-citationThe Evidence Framework Applied to Classification Networks
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.5.720 · Cited by 722
Quantitative analysis of faults and failures in a complex software system
Fenton, N. E., & Ohlsson, N. (2000) · IEEE Transactions on Software Engineering · doi:10.1109/32.879815 · Cited by 699
An Empirical Comparison of Pruning Methods for Decision Tree Induction
Mingers, J. (1989) · Machine Learning · doi:10.1023/a:1022604100933 · Cited by 627
Large Margin Classification Using the Perceptron Algorithm
Freund, Y., & Schapire, R. E. (1999) · Machine Learning · doi:10.1023/a:1007662407062 · Cited by 604
Data mining and knowledge discovery in databases
Fayyad, U., & Uthurusamy, R. (1996) · Communications of the ACM · doi:10.1145/240455.240463 · Cited by 590
The detection of fault-prone programs
Munson, J. C., & Khoshgoftaar, T. M. (1992) · IEEE Transactions on Software Engineering · doi:10.1109/32.135775 · Cited by 442
The prediction of faulty classes using object-oriented design metrics
El Emam, K., Melo, W., & Machado, J. C. (2001) · Journal of Systems and Software · doi:10.1016/s0164-1212(00)00086-8 · Cited by 359
Assessing the applicability of fault-proneness models across object-oriented software projects
Briand, L. C., Melo, W. L., & Wust, J. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1019484 · Cited by 352
Methodology for validating software metrics
Schneidewind, N. F. (1992) · IEEE Transactions on Software Engineering · doi:10.1109/32.135774 · Cited by 351
Predicting fault-prone software modules in telephone switches
Ohlsson, N., & Alberg, H. (1996) · IEEE Transactions on Software Engineering · doi:10.1109/32.553637 · Cited by 346
Comparing software prediction techniques using simulation
Shepperd, M., & Kadoda, G. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.965341 · Cited by 280
Learning from examples: generation and evaluation of decision trees for software resource analysis
Selby, R. W., & Porter, A. A. (1988) · IEEE Transactions on Software Engineering · doi:10.1109/32.9061 · Cited by 279
Menzies, T., Dekhtyar, A., Distefano, J., & Greenwald, J. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.70721 · Cited by 277
Robust Prediction of Fault-Proneness by Random Forests
Guo, L., Ma, Y., Cukic, B., & Singh, H. (2005) · 15th International Symposium on Software Reliability Engineering · doi:10.1109/issre.2004.35 · Cited by 271
Reliability and validity in comparative studies of software prediction models
Myrtveit, I., Stensrud, E., & Shepperd, M. (2005) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2005.58 · Cited by 259
Khoshgoftaar, T. M., Allen, E. B., Hudepohl, J. P., & Aud, S. J. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.595888 · Cited by 198
A controlled experiment to assess the benefits of estimating with analogy and regression models
Myrtveit, I., & Stensrud, E. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799947 · Cited by 183
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/15324430152748218 · Cited by 182
Briand, L. C., Brasili, V. R., & Hetmanski, C. J. (1993) · IEEE Transactions on Software Engineering · doi:10.1109/32.256851 · Cited by 179
Analyzing software measurement data with clustering techniques
Zhong, S., Khoshgoftaar, T. M., & Seliya, N. (2004) · IEEE Intelligent Systems · doi:10.1109/mis.2004.1274907 · Cited by 177
Comparative Assessment of Software Quality Classification Techniques: An Empirical Case Study
Khoshgoftaar, T. M., & Seliya, N. (2004) · Empirical Software Engineering · doi:10.1023/b:emse.0000027781.18360.9b · Cited by 172
Comparing case-based reasoning classifiers for predicting high risk software components
El Emam, K., Benlarbi, S., Goel, N., & Rai, S. N. (2001) · Journal of Systems and Software · doi:10.1016/s0164-1212(00)00079-0 · Cited by 160
A flexible method for software effort estimation by analogy
Li, J., Ruhe, G., Al‐Emran, A., & Richter, M. M. (2006) · Empirical Software Engineering · doi:10.1007/s10664-006-7552-4 · Cited by 156
Comments on "Data Mining Static Code Attributes to Learn Defect Predictors"
Zhang, H., & Zhang, X. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.70706 · Cited by 140
A Second Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems
Grbac, T. G., Runeson, P., & Huljenić, D. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.46 · Cited by 138
A Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems
Andersson, C., & Runeson, P. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.1005 · Cited by 135
A Priori Tests in Repeated Measures Designs: Effects of Nonsphericity
Boik, R. J. (1981) · Psychometrika · doi:10.1007/bf02293733 · Cited by 132
Mining software repositories for comprehensible software fault prediction models
Vandecruys, O., Martens, D., Baesens, B., Mues, C., Backer, M. D., & Haesen, R. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2007.07.034 · Cited by 131
Self-citationAn investigation of the effect of module size on defect prediction using static measures
Koru, A. G., & Liu, H. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083172 · Cited by 92
An empirical study of predicting software faults with case-based reasoning
Khoshgoftaar, T. M., Seliya, N., & Sundaresh, N. (2006) · Software Quality Journal · doi:10.1007/s11219-006-7597-z · Cited by 89
Classification-tree models of software-quality over multiple releases
Khoshgoftaar, T. M., Allen, E. B., Jones, W. D., & HUDEPOHL, J. P. (2000) · IEEE Transactions on Reliability · doi:10.1109/24.855532 · Cited by 89
Assessing Predictors of Software Defects
Menzies, T., Stefano, J. S. D., & Orrego, A. (2004) · Cited by 88
Application of neural networks for predicting program faults
Khoshgoftaar, T. M., Pandya, A. S., & Lanning, D. L. (1995) · Annals of Software Engineering · doi:10.1007/bf02249049 · Cited by 66
Evaluating predictive quality models derived from software measures: Lessons learned
Lanubile, F., & Visaggio, G. (1997) · Journal of Systems and Software · doi:10.1016/s0164-1212(96)00153-7 · Cited by 62
A replicated empirical study of a selection method for software reliability growth models
Andersson, C. (2006) · Empirical Software Engineering · doi:10.1007/s10664-006-9018-0 · Cited by 51
Cited by
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154
An experimental comparison of classification algorithms for imbalanced credit scoring data sets
Brown, I., & Mues, C. (2012) · Expert Systems with Applications · doi:10.1016/j.eswa.2011.09.033 · Cited by 730
Self-citationOn the relative value of cross-company and within-company data for defect prediction
Turhan, B., Menzies, T., Bener, A. B., & Di Stefano, J. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9103-7 · Cited by 725
A large-scale empirical study of just-in-time quality assurance
Kamei, Y., Shihab, E., Adams, B., Hassan, A. E., Mockus, A., Sinha, A., & Ubayashi, N. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.70 · Cited by 715
An exploratory study of the pull-based software development model
Gousios, G., Pinzger, M., & Deursen, A. V. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568260 · Cited by 628
Metadata
- Authors
- Lessmann, S., Baesens, B., Mues, C., & Pietsch, S.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2008-07
- Volume
- 34
- Issue
- 4
- Pages
- 485-496
- ISSN
- 0098-5589 (ISSN-L)
- OpenAlex
- W2105776892
- Cited by
- 1,226
- References
- 66
- FWCI
- 123.30
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.