An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
The DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index.
Abstract
Defect prediction models help software quality assurance teams to allocate their limited resources to the most defect-prone modules. Model validation techniques, such as$k$-fold cross-validation, use historical data to estimate how well a model will perform in the future. However, little is known about how accurate the estimates of model validation techniques tend to be. In this paper, we investigate the bias and variance of model validation techniques in the domain of defect prediction. Analysis of 101 public defect datasets suggests that 77 percent of them are highly susceptible to producing unstable results– - selecting an appropriate model validation technique is a critical experimental design choice. Based on an analysis of 256 studies in the defect prediction literature, we select the 12 most commonly adopted model validation techniques for evaluation. Through a case study of 18 systems, we find that single-repetition holdout validation tends to produce estimates with 46-229 percent more bias and 53-863 percent more variance than the top-ranked model validation techniques. On the other hand, out-of-sample bootstrap validation yields the best balance between the bias and variance of estimates in the context of our study. Therefore, we recommend that future defect prediction studies avoid single-repetition holdout validation, and instead, use out-of-sample bootstrap validation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Biases may be balanced with variance of performance estimates using out-of-sample bootstrap techniques [139].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [139]
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
An Analysis of Transformations
Box, G. E. P., & Cox, D. R. (1964) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1964.tb00553.x · Cited by 15,052
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655
Building Predictive Models in R Using the caret Package
Kuhn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 8,863
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951 · cited 1×
“On the other hand, out-of-sample bootstrap validation yields the best balance between the bias and variance of estimates in the context of our study.”
BackgroundHarrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
A leisurely look at the bootstrap, the jackknife, and cross-validation
Efron, B., & Gong, G. (1983) · The American Statistician · doi:10.1080/00031305.1983.10483087 · Cited by 3,229
Regression Modeling Strategies
Walker, E. (2003) · Technometrics · doi:10.1198/tech.2003.s158 · Cited by 3,188
On the Optimality of the Simple Bayesian Classifier under Zero-One Loss
Domingos, P., & Pazzani, M. (1997) · Machine Learning · doi:10.1023/a:1007413511361 · Cited by 3,101
A Cluster Analysis Method for Grouping Means in the Analysis of Variance
Scott, A. J., & Knott, M. (1974) · Biometrics · doi:10.2307/2529204 · Cited by 2,770
The Predictive Sample Reuse Method with Applications
Geisser, S. (1975) · Journal of the American Statistical Association · doi:10.1080/01621459.1975.10479865 · Cited by 2,748
Estimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300
caret: Classification and Regression Training
Kuhn, M. (2007) · CRAN: Contributed Packages · doi:10.32614/cran.package.caret · Cited by 1,786
Improvements on Cross-Validation: The 632+ Bootstrap Method
Efron, B., & Tibshirani, R. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474007 · Cited by 1,636
Prediction error estimation: a comparison of resampling methods
Molinaro, A. M., Simon, R., & Pfeiffer, R. M. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti499 · Cited by 1,339
Lessmann, S., Baesens, B., Mues, C., & Pietsch, S. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.35 · Cited by 1,226
Data Mining Static Code Attributes to Learn Defect Predictors
Menzies, T., Greenwald, J., & Frank, A. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.256941 · Cited by 1,215
A Leisurely Look at the Bootstrap, the Jackknife, and Cross-Validation
Efron, B., & Gong, G. D. (1983) · The American Statistician · doi:10.2307/2685844 · Cited by 1,160
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154 · cited 1×
“Thus, we select a sample from the JM1 dataset such that the EPV is 3.”
MethodDangers of Using "Optimal" Cutpoints in the Evaluation of Prognostic Factors
Altman, D. G., Lausen, B., Sauerbrei, W., & Schumacher, M. (1994) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/86.11.829 · Cited by 1,143
The Risk of Determining Risk with Multivariable Models
Concato, J., Feinstein, A. R., & Holford, T. R. (1993) · Annals of Internal Medicine · doi:10.7326/0003-4819-118-3-199302010-00009 · Cited by 1,115
Arcuri, A., & Briand, L. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985795 · Cited by 968
Improving your data transformations: Applying the Box-Cox transformation
Osborne, J. W. (2020) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/qbpc-gk17 · Cited by 878
Steyerberg, E. W., Eijkemans, M. J. C., Harrell, F. E., & Habbema, J. D. F. (2000) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(20000430)19:8<1081::aid-sim406>3.0.co;2-a · Cited by 802
Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap
Kim, J.-H. (2009) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2009.04.009 · Cited by 792
Steyerberg, E. W., Eijkemans, M. J. C., Harrell, F. E., & Habbema, J. D. F. (2000) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(20000430)19:8<1059::aid-sim412>3.0.co;2-0 · Cited by 785
Use of relative code churn measures to predict system defect density
Nagappan, N., & Ball, T. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062514 · Cited by 775 · cited 1×
“In addition to the discrimination power, practitioners often use the predicted probabilities to rank the defectprone files [74], [79], [96], [122].”
BackgroundPredicting Defects for Eclipse
Zimmermann, T., Premraj, R., & Zeller, A. (2007) · Third International Workshop on Predictor Models in Software Engineering (PROMISE'07: ICSE Workshops 2007) · doi:10.1109/promise.2007.10 · Cited by 753
Beanplot: A Boxplot Alternative for Visual Comparison of Distributions
Kampstra, P. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.c01 · Cited by 734
Improvements on Cross-Validation: The .632+ Bootstrap Method
Efron, B., & Tibshirani, R. J. (1997) · Journal of the American Statistical Association · doi:10.2307/2965703 · Cited by 723
The Basic Practice of Statistics
Moore, L. M. (1996) · Technometrics · doi:10.1080/00401706.1996.10484558 · Cited by 719
Cross-project defect prediction: a large scale experiment on data vs. domain vs. process
Zimmermann, T., Nagappan, N., Gall, H. C., Giger, E., & Murphy, B. (2009) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-25785 · Cited by 694
The PROMISE Repository of Empirical Software Engineering Data
Boetticher, G. D. (2007) · Medical Entomology and Zoology · Cited by 682
Submodel Selection and Evaluation in Regression. The X-Random Case
Breiman, L., & Spector, P. (1992) · International Statistical Review / Revue Internationale de Statistique · doi:10.2307/1403680 · Cited by 669
Bias plus variance decomposition for zero-one loss functions
Kohavi, R., & Wolpert, D. H. (1996) · Cited by 652
Artificial Neural Networks: An Introduction
Priddy, K. L., & Keller, P. E. (2005) · SPIE eBooks · doi:10.1117/3.633187 · Cited by 639
An extensive comparison of bug prediction approaches
D'Ambros, M., Lanza, M., & Robbes, R. (2010) · 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010) · doi:10.1109/msr.2010.5463279 · Cited by 637
Evaluating defect prediction approaches: a benchmark and an extensive comparison
D’Ambros, M., Lanza, M., & Robbes, R. (2012) · Empirical Software Engineering · doi:10.1007/s10664-011-9173-9 · Cited by 626
Is cross-validation valid for small-sample microarray classification?
Braga-Neto, U. M., & Dougherty, E. R. (2004) · Bioinformatics · doi:10.1093/bioinformatics/btg419 · Cited by 597
How Biased is the Apparent Error Rate of a Prediction Rule?
Efron, B. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478291 · Cited by 587
Clinical Prediction Models—a Practical Approach to Development, Validation and Updating
Bedogni, G. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2009.00614_16.x · Cited by 581
Predicting defects using network analysis on dependency graphs
Zimmermann, T., & Nagappan, N. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368161 · Cited by 580
Data Quality: Some Comments on the NASA Software Defect Datasets
Shepperd, M., Song, Q., Sun, Z., & Mair, C. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.11 · Cited by 570
Two Further Applications of a Model for Binary Regression
Cox, D. R. (1958) · Biometrika · doi:10.2307/2333203 · Cited by 557
Two further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556
Towards identifying software project clusters with regard to defect prediction
Jureczko, M., & Madeyski, L. (2010) · Proceedings of the 6th International Conference on Predictive Models in Software Engineering · doi:10.1145/1868328.1868342 · Cited by 546
Austin, P. C., & Steyerberg, E. W. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280214558972 · Cited by 516 · cited 1×
“Broadly speaking, a defect prediction model is a statistical regression model or a machine learning classifier that is trained to identify defect-prone software modules.”
BackgroundEnsemble Methods in Data Mining: Improving Accuracy Through Combining Predictions
Seni, G., & Elder, J. F. (2010) · Synthesis lectures on data mining and knowledge discovery · doi:10.2200/s00240ed1v01y200912dmk002 · Cited by 492
Lee, K. L., & Mark, D. B. (1996) · Cited by 488
Transfer learning for cross-company software defect prediction
Ma, Y., Luo, G., Zeng, X., & Chen, A. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2011.09.007 · Cited by 475
Arisholm, E., Briand, L. C., & Johannessen, E. B. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2009.06.055 · Cited by 455
Use of Brier score to assess binary predictions
Rufibach, K. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.11.009 · Cited by 454
Predicting risk of software changes
Mockus, A., & Weiss, D. M. (2002) · Bell Labs Technical Journal · doi:10.1002/bltj.2229 · Cited by 429
ReLink: recovering links between bugs and changes
Wu, R., Zhang, H., Kim, S., & Cheung, S.-C. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025120 · Cited by 412
A General Software Defect-Proneness Prediction Framework
Song, Q., Jia, Z., Shepperd, M., Ying, S., & Liu, J. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.90 · Cited by 374
Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.91 · Cited by 364
Self-citationThe influence of organizational structure on software quality: an empirical case study
Nagappan, N., Murphy, B., & Basili, V. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368160 · Cited by 363
Don't touch my code!: examining the effects of ownership on software quality
Bird, C., Nagappan, N., Murphy, B., Gall, H., & Devanbu, P. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025119 · Cited by 340
Automated parameter optimization of classification techniques for defect prediction models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884857 · Cited by 331
Self-citationDealing with noise in defect prediction
Kim, S., Zhang, H., Wu, R., & Gong, L. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985859 · Cited by 330
Validation techniques for logistic regression models
Miller, M. E., Hui, S. L., & Tierney, W. M. (1991) · Statistics in Medicine · doi:10.1002/sim.4780100805 · Cited by 318
Comparing software prediction techniques using simulation
Shepperd, M., & Kadoda, G. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.965341 · Cited by 280
Robust Prediction of Fault-Proneness by Random Forests
Guo, L., Ma, Y., Cukic, B., & Singh, H. (2005) · 15th International Symposium on Software Reliability Engineering · doi:10.1109/issre.2004.35 · Cited by 271
Techniques for evaluating fault prediction models
Jiang, Y., Cukic, B., & Ma, Y. (2008) · Empirical Software Engineering · doi:10.1007/s10664-008-9079-3 · Cited by 267
Reliability and validity in comparative studies of software prediction models
Myrtveit, I., Stensrud, E., & Shepperd, M. (2005) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2005.58 · Cited by 259
Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2015) · Cited by 256
Self-citationHow, and why, process metrics are better
Rahman, F., & Devanbu, P. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606589 · Cited by 248
Recalling the "imprecision" of cross-project defect prediction
Rahman, F., Posnett, D., & Devanbu, P. (2012) · Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering · doi:10.1145/2393596.2393669 · Cited by 246
Ranking and Clustering Software Cost Estimation Models through a Multiple Comparisons Algorithm
Mittas, N., & Angelis, L. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.45 · Cited by 234
Online Defect Prediction for Imbalanced Data
Tan, M., Tan, L., Dara, S., & Mayeux, C. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.139 · Cited by 229
Data mining techniques for software effort estimation: a comparative study
Dejaeger, K., Verbeke, W., Martens, D., & Baesens, B. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.55 · Cited by 224
Effort-Aware Defect Prediction Models
Mende, T., & Koschke, R. (2010) · 2010 14th European Conference on Software Maintenance and Reengineering · doi:10.1109/csmr.2010.18 · Cited by 209
Cross-validation and bootstrapping are unreliable in small sample classification
Isaksson, A., Wallman, M., Göransson, H., & Gustafsson, M. G. (2008) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.06.018 · Cited by 186
How, and Why, Process Metrics are better
Rahman, F., & Dévanbu, P. (2013) · Cited by 184
An industrial study on the risk of software changes
Shihab, E., Hassan, A. E., Adams, B., & Jiang, Z. M. (2012) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2393596.2393670 · Cited by 174
Self-citationComparative Assessment of Software Quality Classification Techniques: An Empirical Case Study
Khoshgoftaar, T. M., & Seliya, N. (2004) · Empirical Software Engineering · doi:10.1023/b:emse.0000027781.18360.9b · Cited by 172
On Parameter Tuning in Search Based Software Engineering
Arcuri, A., & Fraser, G. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-23716-4_6 · Cited by 170
The Impact of Mislabelling on the Performance and Interpretation of Defect Prediction Models
Tantithamthavorn, C. K., McIntosh, S., Hassan, A. E., Ihara, A., & Matsumoto, K. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.93 · Cited by 169
Self-citationeffsize: Efficient Effect Size Computation
Torchiano, M. (2013) · CRAN: Contributed Packages · doi:10.32614/cran.package.effsize · Cited by 168
Software effort models should be assessed via leave-one-out validation
Kocagüneli, E., & Menzies, T. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2013.02.053 · Cited by 158
Online defect prediction for imbalanced data
Tan, M., Tan, L., Dara, S., & Mayeux, C. (2015) · doi:10.5555/2819009.2819026 · Cited by 151
Balancing Privacy and Utility in Cross-Company Defect Prediction
Peters, F., Menzies, T., Gong, L., & Zhang, H. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.6 · Cited by 139
On the dataset shift problem in software engineering prediction models
Turhan, B. (2011) · Empirical Software Engineering · doi:10.1007/s10664-011-9182-8 · Cited by 132
A Comparative Study of Ensemble Feature Selection Techniques for Software Defect Prediction
Wang, H., Khoshgoftaar, T. M., & Napolitano, A. (2010) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2010.27 · Cited by 123
Fault Prediction using Early Lifecycle Data
Jiang, Y., Čukić, B., & Menzies, T. (2007) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2007.24 · Cited by 121
Empirical evaluation of the effects of mixed project data on learning defect predictors
Turhan, B., Tosun, A., & Bener, A. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.10.003 · Cited by 120
Sample size vs. bias in defect prediction
Rahman, F., Posnett, D., Herraiz, I., & Dévanbu, P. (2013) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2491411.2491418 · Cited by 117
Naive Bayes Software Defect Prediction Model
Wang, T., & Li, W. (2010) · International Conference on Computational Intelligence and Software Engineering (CiSE) · doi:10.1109/cise.2010.5677057 · Cited by 112
Special issue on repeatable results in software engineering prediction
Menzies, T., & Shepperd, M. (2012) · Empirical Software Engineering · doi:10.1007/s10664-011-9193-5 · Cited by 107
Can data transformation help in the detection of fault-prone modules?
Jiang, Y., Čukić, B., & Menzies, T. (2008) · doi:10.1145/1390817.1390822 · Cited by 104
Performance of prognostic models in critically ill cancer patients – a review
den Boer, S. D., de Keizer, N. D., & de Jonge, E. (2005) · Critical Care · doi:10.1186/cc3765 · Cited by 99
An empirical evaluation of ensemble adjustment methods for analogy-based effort estimation
Azzeh, M., Nassif, A. B., & Minku, L. L. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2015.01.028 · Cited by 97
High-impact defects: a study of breakage and surprise defects
Shihab, E., Mockus, A., Kamei, Y., Adams, B., & Hassan, A. E. (2011) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2025113.2025155 · Cited by 96
Self-citationVariance Analysis in Software Fault Prediction Models
Jiang, Y., Lin, J., Čukić, B., & Menzies, T. (2009) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2009.13 · Cited by 94
The impact of parameter tuning on software effort estimation using learning machines
Song, L., Minku, L. L., & Yao, X. (2013) · doi:10.1145/2499393.2499394 · Cited by 86
A Comprehensive Empirical Study of Count Models for Software Fault Prediction
Gao, K., & Khoshgoftaar, T. M. (2007) · IEEE Transactions on Reliability · doi:10.1109/tr.2007.896761 · Cited by 82
Variance reduction in estimating classification error using sparse datasets
Beleites, C., Baumgartner, R., Bowman, C., Somorjai, R. L., Steiner, G. R., Salzer, R., & Sowa, M. G. (2005) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2005.04.008 · Cited by 79
Predicting Subsystem Failures using Dependency Graph Complexities
Zimmermann, T., & Nagappan, N. (2007) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2007.19 · Cited by 78
Artificial Neural Networks: An Introduction (SPIE Tutorial Texts in Optical Engineering, Vol. TT68)
Priddy, K. L., & Keller, P. E. (2005) · Cited by 77
Comments on “Researcher Bias: The Use of Machine Learning in Software Defect Prediction”
Tantithamthavorn, C. K., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2553030 · Cited by 76
Self-citationPerformance of Error Estimators for Classification
Dougherty, E. R., Sima, C., Hua, >., Hanczar, B., & Braga-Neto, U. (2010) · Current Bioinformatics · doi:10.2174/157489310790596385 · Cited by 59
How effective is Tabu search to configure support vector regression for effort estimation?
Corazza, A., Martino, S. D., Ferrucci, F., Gravino, C., Sarro, F., & Mendes, E. (2010) · doi:10.1145/1868328.1868335 · Cited by 59
Replication of defect prediction studies
Mende, T. (2010) · doi:10.1145/1868328.1868336 · Cited by 58
Cross-project defect prediction: a large scale experiment on data vs. domain vs. process
Zimmermann, T., Nagappan, N., Gall, H. C., Giger, E., & Murphy, T. B. (2009) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1595696.1595713 · Cited by 57
Validity and reliability of evaluation procedures in comparative studies of effort prediction models
Myrtveit, I., & Stensrud, E. (2011) · Empirical Software Engineering · doi:10.1007/s10664-011-9183-7 · Cited by 57
Data mining source code for locating software bugs: A case study in telecommunication industry
Turhan, B., Kocak, G., & Bener, A. (2008) · Expert Systems with Applications · doi:10.1016/j.eswa.2008.12.028 · Cited by 57
Adequate and Precise Evaluation of Quality Models in Software Engineering Studies
Ma, Y., & Čukić, B. (2007) · Predictive Models in Software Engineering · doi:10.1109/promise.2007.1 · Cited by 46
Ensemble of software defect predictors
Tosun, A., Turhan, B., & Bener, A. (2008) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1414004.1414066 · Cited by 40
An Exploration of Challenges Limiting Pragmatic Software Defect Prediction
Shihab, E. (2012) · QSpace (Queen's University Library) · Cited by 39
Tantithamthavorn, C. K. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2889160.2889256 · Cited by 28
Self-citationDavison, A. C., & Hall, P. M. (1992) · Biometrika · doi:10.1093/biomet/79.2.279 · Cited by 22
INCREMENTAL DEVELOPMENT OF FAULT PREDICTION MODELS
Jiang, Y., Čukić, B., Menzies, T., & Lin, J. (2013) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194013500447 · Cited by 9
Using Statistical Models to Predict Software Regressions
Tarvo, A. (2008) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2008.21 · Cited by 6
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
What do we mean by validating a prognostic model?
(2000) · doi:10.1002/(sici)1097-0258(20000229)19:4<453::aid-sim350>3.0.co;2-5
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4<361::aid-sim168>3.0.co;2-4
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 1×
“We use the Brier score [14], [89] to measure the distance between the predicted probabilities and the outcome.”
Method
Research works printed but not linked: 9
- 10.29172/7c2a6982-6d72-4cd8-bba6-2fccb06a7011 · DOI · Paper page
- 10.2307/2289236 · doi:10.2307/2289236 · Paper page
- Jelihovschi, 2014, The ScottKnott Clustering Algorithm
- 10.1067/mod.2000.109032 · DOI · Paper page
- Meyer, 2014, E1071: Misc functions of the department of statistics (E1071), TU Wien
- 10.1109/ICSE.2005.1553571 · DOI · Paper page
- 2013, R Core Team, R: A language and environment for statistical computing, R foundation for statistical computing, Vienna, Austria
- 10.1007/978-3-031-01899-2 · doi:10.1007/978-3-031-01899-2 · Paper page
- Tantithamthavorn, 2015, ScottKnottESD: An R package of the Scott-Knott effect size difference (ESD) test
Cited by
A Novel Neural Source Code Representation Based on Abstract Syntax Tree
Zhang, J., Wang, X., Zhang, H., Sun, H., Wang, K., & Liu, X. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00086 · Cited by 655
“Many software engineering methods, such as source code classification [1], [2], code clone detection [3], [4], [5], [6], defect prediction [7], [8] and code summarization [9], [10] have been proposed to improve software development and maintenance.”
MethodThe Impact of Automated Parameter Optimization on Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2794977 · Cited by 415
“Our recent work shows that defect prediction models that are trained using datasets with low EPV values are especially susceptible to unstable results [141].”
BackgroundMethodInfluentialSelf-citationAutomated parameter optimization of classification techniques for defect prediction models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884857 · Cited by 331
Self-citationA framework for data-driven digital twins of smart manufacturing systems
Friederich, J., Francis, D. P., Lazarova-Molnar, S., & Mohamed, N. (2022) · Computers in Industry · doi:10.1016/j.compind.2021.103586 · Cited by 323
“It has been proposed that out-of-sample bootstrap techniques tend to give estimates that are the least biased (Tantithamthavorn et al., 2017).”
BackgroundTantithamthavorn, C., Hassan, A. E., & Matsumoto, K. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876537 · Cited by 327
Self-citation
Metadata
- Authors
- Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K.
- Affiliations
- Nara Institute of Science and Technology (JP)
- Queen's University (CA)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2017-01-01
- Volume
- 43
- Issue
- 1
- Pages
- 1-18
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2474835145
- Cited by
- 574
- References
- 129
- Citations by year
- 2026: 39 · 2025: 53 · 2024: 69 · 2023: 56 · 2022: 58 · 2021: 93 · 2020: 63 · 2019: 75 · 2018: 45 · 2017: 18
- FWCI
- 106.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Funders
- JSPS
- Advancing Strategic International Networks to Accelerate the Circulation of Talented Researchers
- Interdisciplinary Global Networks for Accelerating Theory and Practice in Software Ecosystem
- JSPS Fellows (16J03360)
- Natural Sciences and Engineering Research Council of Canada (NSERC)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- fault prediction, model validation, k-fold cross-validation, software quality assurance, defect dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.