An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
The DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index.
Abstract
Defect prediction models help software quality assurance teams to allocate their limited resources to the most defect-prone modules. Model validation techniques, such as$k$-fold cross-validation, use historical data to estimate how well a model will perform in the future. However, little is known about how accurate the estimates of model validation techniques tend to be. In this paper, we investigate the bias and variance of model validation techniques in the domain of defect prediction. Analysis of 101 public defect datasets suggests that 77 percent of them are highly susceptible to producing unstable results– - selecting an appropriate model validation technique is a critical experimental design choice. Based on an analysis of 256 studies in the defect prediction literature, we select the 12 most commonly adopted model validation techniques for evaluation. Through a case study of 18 systems, we find that single-repetition holdout validation tends to produce estimates with 46-229 percent more bias and 53-863 percent more variance than the top-ranked model validation techniques. On the other hand, out-of-sample bootstrap validation yields the best balance between the bias and variance of estimates in the context of our study. Therefore, we recommend that future defect prediction studies avoid single-repetition holdout validation, and instead, use out-of-sample bootstrap validation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Biases may be balanced with variance of performance estimates using out-of-sample bootstrap techniques [139].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [139]
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
An Analysis of Transformations
Box, G. E. P., & Cox, D. R. (1964) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1964.tb00553.x · Cited by 15,052
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Research works printed but not linked: 9
- 10.29172/7c2a6982-6d72-4cd8-bba6-2fccb06a7011 · DOI · Paper page
- 10.2307/2289236 · doi:10.2307/2289236 · Paper page
- Jelihovschi, 2014, The ScottKnott Clustering Algorithm
- 10.1067/mod.2000.109032 · DOI · Paper page
- Meyer, 2014, E1071: Misc functions of the department of statistics (E1071), TU Wien
- 10.1109/ICSE.2005.1553571 · DOI · Paper page
- 2013, R Core Team, R: A language and environment for statistical computing, R foundation for statistical computing, Vienna, Austria
- 10.1007/978-3-031-01899-2 · doi:10.1007/978-3-031-01899-2 · Paper page
- Tantithamthavorn, 2015, ScottKnottESD: An R package of the Scott-Knott effect size difference (ESD) test
Cited by
Items 201–300 of 575
csDetector: an open source tool for community smells detection
Almarimi, N., Ouni, A., Chouchen, M., & Mkaouer, M. W. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473121 · Cited by 31
“times since it was showed to be stable for similar software engineering problems [39].”
BackgroundChen, H., Jing, X., Zhou, Y., Li, B., & Xu, B. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106892 · Cited by 30
Context- and Fairness-Aware In-Process Crowdworker Recommendation
Wang, J., Yang, Y., Wang, S., Hu, J., & Wang, Q. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487571 · Cited by 30
“As reported by existing work that unbalanced data could significantly affect the model performance [69, 70], to make our dataset balanced, we randomly sample an equal number of crowdworkers (who didn’t submit bugs in the specific task) with the positive instances and label them as 0.”
Odejide, B. J., Bajeh, A. O., Balogun, A. O., Alanamu, Z. O., Adewole, K. S., Akintola, A. G., Salihu, S. A., Usman-Hamza, F. E., & Mojeed, H. A. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-09070-7_49 · Cited by 28
Eigenvalue Distributions in Random Confusion Matrices: Applications to Machine Learning Evaluation
Olaniran, O. R., Alzahrani, A., & Alzahrani, M. R. (2024) · Mathematics · doi:10.3390/math12101425 · Cited by 27
Identifying gameplay videos that exhibit bugs in computer games
Lin, D., Bezemer, C., & Hassan, A. E. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09733-6 · Cited by 34
“Step A1 - Evaluating the classifiers To compare the performance of the classifiers to the na¨ıve keyword search approach and each other, we used the out-of-sample bootstrap validation technique [42].”
MethodSelf-citationConvolutional Support Vector Models: Prediction of Coronavirus Disease Using Chest X-rays
Maia, M., Pimentel, J. S., Pereira, I. S., Gondim, J. A. M., Barreto, M. E., & Ara, A. (2020) · Information · doi:10.3390/info11120548 · Cited by 27
Exploiting gated graph neural network for detecting and explaining self-admitted technical debts
Yu, J., Zhao, K., Liu, J., Liu, X., Zhou, X., & Wang, X. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111219 · Cited by 30
An Efficient Software Defect Analysis Using Correlation-Based Oversampling
Rao, K. N., & Reddy, C. S. (2018) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-018-3076-7 · Cited by 24
Machine Learning models to predict Agile Methodology adoption
Hanslo, R., & Tanner, M. (2020) · Annals of Computer Science and Information Systems · doi:10.15439/2020f214 · Cited by 31
“Firstly, the training and test split used in this research paper for the dataset has been reported previously as displaying biases [24].”
MethodPredicting Community Smells’ Occurrence on Individual Developers by Sentiments
Huang, Z., Shao, Z., Fan, G., Gao, J., Zhou, Z., Yang, K., & Yang, X. (2021) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc52881.2021.00030 · Cited by 23
FENSE: A feature-based ensemble modeling approach to cross-project just-in-time defect prediction
Zhang, T., Yu, Y., Mao, X., Lu, Y., Li, Z., & Wang, H. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10185-8 · Cited by 22
“We apply a Scott-Knott ESD test (Tantithamthavorn et al. 2017, Tantithamtha-vorn et al. 2018) as our techniques for the comparison test.”
MethodWhat makes Ethereum blockchain transactions be processed fast or slow? An empirical study
Pacheco, M., Oliva, G. A., Rajbahadur, G. K., & Hassan, A. E. (2023) · Empirical Software Engineering · doi:10.1007/s10664-022-10283-7 · Cited by 30
“Furthermore, we also log transform all the independent features of the type numeric with an identical function to reduce the bias caused by outliers, similar to prior studies (Lee et al. 2020; Mcintosh et al. 2016; Menzies et al. 2007; Tantithamthavorn et al. 2016).”
ResultSelf-citationWhat Makes a High-Quality Training Dataset for Large Language Models: A Practitioners' Perspective
Yu, X., Zhang, Z., Niu, F., Hu, X., Xia, X., & Grundy, J. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695061 · Cited by 71
“In line with practices in previous studies [66, 75, 76], we employ the Scott-Knott Effect Size Difference (SKESD) test [68] to divide the 18 characteristics into different groups based on their Likert scores.”
The Impact of Data Merging on the Interpretation of Cross-Project Just-In-Time Defect Models
Lin, D., Tantithamthavorn, C. K., & Hassan, A. E. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3073920 · Cited by 28
Self-citationEnhancing the defectiveness prediction of methods and classes via JIT
Falessi, D., Laureani, S. M., Çarka, J., Esposito, M., & da Costa, D. A. (2023) · Empirical Software Engineering · doi:10.1007/s10664-022-10261-z · Cited by 29
“The manner in which defects are introduced into code, and the sheer volume of defects in software, are typically beyond the capability and resources of most development teams (Tantithamthavorn et al. 2016a; Nam et al. 2017; Kamei et al. 2012; Ghotra et al. 2017; Kondo et al. 2019).”
BackgroundMethodAutomatically Recommend Code Updates: Are We There Yet?
Liu, Y., Tantithamthavorn, C. K., Liu, Y., Thongtanunam, P., & Li, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3678167 · Cited by 32
“Extending the concerns related to evaluation techniques [44, 68, 80], this paper is the first to demonstrate the impact of the time-wise evaluation scenario for various CodeLM-based code generation tasks (including our own code update recommendation tool and the automated code transformation tool…”
Self-citationJoint distribution matching model for distribution–adaptation‐based cross‐project defect prediction
Qiu, S., Lu, L., & Jiang, S. (2019) · IET Software · doi:10.1049/iet-sen.2018.5131 · Cited by 21
Improved SAR target recognition by selecting moment methods based on Fisher score
Bolourchi, P., Moradi, M., Demirel, H., & Uysal, S. (2019) · Signal Image and Video Processing · doi:10.1007/s11760-019-01521-5 · Cited by 21
Moving from Closed to Open Source: Observations from Six Transitioned Projects to GitHub
Kochhar, P. S., Kalliamvakou, E., Nagappan, N., Zimmermann, T., & Bird, C. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2937025 · Cited by 26
“Table 10 presents the hypotheses as ranked by the Scott-Knott Effect Size Difference (ESD) test [32] according to their Likert scores.”
BackgroundMethodInfluentialDefect Prediction via LSTM Based on Sequence and Tree Structure
Zhou, X., & Lu, L. (2020) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs51102.2020.00055 · Cited by 25
“In our study, we employed the Scott-Knott effect size difference (ESD) [35], the normality and effect size aware variant of traditional Scott-Knott test.”
MethodJust-in-time defect prediction for Android apps via imbalanced deep learning model
Zhao, K., Xu, Z., Yan, M. L., Tang, Y., Fan, M. Y., & Catolino, G. (2021) · ACM Symposium on Applied Computing · doi:10.1145/3412841.3442019 · Cited by 21
“In this work, we apply a state-of-the-art method, namely ScottKnott Effect Size Difference (short for SKESD) test [27], to analyze the significant differences between our IDL method and the compar-”
MethodEthnobotany Database: Exploring Diversity Medicinal Plants of Dayak Tribe Borneo
Haeruddin, H., Hairah, U., Budiman, E., & Johan, H. (2017) · Proceeding of the Electrical Engineering Computer Science and Informatics · doi:10.11591/eecsi.v4.1001 · Cited by 20
What makes a popular academic AI repository?
Fan, Y., Xia, X., Lo, D., Hassan, A. E., & Li, S. (2021) · Empirical Software Engineering · doi:10.1007/s10664-020-09916-6 · Cited by 43
Self-citationVoting Heterogeneous Ensemble for Code Smell Detection
Aljamaan, H. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00148 · Cited by 20
Huang, Z., Yu, H., Fan, G., Shao, Z., Li, M., & Liang, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121640 · Cited by 27
An Extensive Analysis of Efficient Bug Prediction Configurations
Osman, H., Ghafari, M., Nierstrasz, O., & Lungu, M. F. (2017) · doi:10.1145/3127005.3127017 · Cited by 23
Winning the app production rally
Noei, E., da Costa, D. A., & Zou, Y. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236044 · Cited by 20
“5, which indicates that our model has a low risk of over-fitting [89].”
BackgroundCross-Project Defect Prediction via Transferable Deep Learning-Generated and Handcrafted Features
Qiu, S., Lu, L., Cai, Z., & Jiang, S. (2019) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2019-070 · Cited by 22
“A detailed description of the Scott-Knott ESD test can be seen in [24].”
MethodLocal versus Global Models for Just-In-Time Software Defect Prediction
Yang, X., Yu, H., Fan, G., Shi, K., & Chen, L. (2019) · Scientific Programming · doi:10.1155/2019/2384706 · Cited by 20
Lomio, F., Moreschini, S., & Lenarduzzi, V. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10164-z · Cited by 31
“We had two possible options adopting ScottKnott test (Tantithamthavorn et al. 2017, 2018) or Nemenyi test8 post-hoc test (Nemenyi 1962).”
MethodJust-in-Time Code Duplicates Extraction
Alomar, E., Ivanov, A., Kurbatova, Z., Golubev, Y., Mkaouer, M., Ouni, A., Bryksin, T., Nguyen, L., Kini, A., & Thakur, A. V. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107169 · Cited by 21
“To evaluate the performance of the algorithms, we use out-of-sample bootstrap validation since this validation technique yields the best balance between the bias and variance in comparison to single-repetition holdout validation [59].”
MethodThongtanunam, P., Shang, W., & Hassan, A. E. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9645-2 · Cited by 23
“Prior work also shows that the bootstrap technique provides less biased performance estimates than the commonly-used validation techniques, e.g., 10-fold cross validation (Efron 1983; Harrell FE 2002; Tantithamthavorn et al. 2017).”
MethodSelf-citationNonlinear classification of emotion from EEG signal based on maximized mutual information
Ghosh, S. M., Bandyopadhyay, S., & Mitra, D. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115605 · Cited by 19
Gated Homogeneous Fusion Networks With Jointed Feature Extraction for Defect Prediction
Fang, D., Liu, S., & Ai, L. (2022) · IEEE Transactions on Reliability · doi:10.1109/tr.2022.3165115 · Cited by 19
ARRAY: Adaptive triple feature-weighted transfer Naive Bayes for cross-project defect prediction
Tong, H., Lu, W., Xing, W., & Wang, S. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111721 · Cited by 22
Binkhonain, M., & Alfayez, R. (2025) · Requirements Engineering · doi:10.1007/s00766-025-00451-8 · Cited by 38
An Empirical Study on the Code Refactoring Capability of Large Language Models
Cordeiro, J., Noei, S., & Zou, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3801158 · Cited by 94
“To compare different prompting techniques and assess whether the prompting techniques lead to statistically significant improvements, we perform the Scott-Knott ESD test [73, 74] across the three distributions: zero-shot, one-shot, and chain-of-thought for each LLM.”
InfluentialSlice-Based Cognitive Complexity Metrics for Defect Prediction
Alqadi, B. S., & Maletic, J. I. (2020) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner48275.2020.9054836 · Cited by 21
“…EPV: Prior studies show that the Events Per Variable (EPV) (i.e., the ratio of the frequency of the least occurring class in the outcome variable to the number of features that are involved in training of a classifier) has a significant influence on the performance of defect classifiers [64].”
BackgroundMethodResultInfluentialAn empirical comparison of validation methods for software prediction models
Ali, A., & Gravino, C. (2021) · Journal of Software Evolution and Process · doi:10.1002/smr.2367 · Cited by 28
“Similarly, Falessi et al. 5 compare the time series validation method with the non-time-series validation method and state that the accuracy can significantly change when different validation methods are used in the domain of SFP. Tantithamthavorn et al. in their empirical study in the context of SFP 3 investigate dif…”
BackgroundMethodInfluentialWhere Should I Look at? Recommending Lines that Reviewers Should Pay Attention To
Hong, Y., Tantithamthavorn, C. K., & Thongtanunam, P. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00121 · Cited by 27
Self-citationTowards Privacy Preserving Cross Project Defect Prediction with Federated Learning
Yamamoto, H., Wang, D., Rajbahadur, G. K., Kondo, M., Kamei, Y., & Ubayashi, N. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00052 · Cited by 25
Adversarial domain adaptation for cross-project defect prediction
Song, H., Wu, G., Ma, L., Pan, Y., Huang, Q., & Jiang, S. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10371-2 · Cited by 15
“Besides, we utilize a variant of Scott-Knott Effect Size Difference (ESD) test (Tantithamthavorn et al. 2016) to measure the effect size between two models and rank all compared models.”
MethodLineFlowDP: A Deep Learning-Based Two-Phase Approach for Line-Level Defect Prediction
Yang, F., Zhong, F., Zeng, G., Xiao, P., & Zheng, W. (2024) · Empirical Software Engineering · doi:10.1007/s10664-023-10439-z · Cited by 21
“Then we apply the ScottKnott ESD test to cluster the distributions into statistically distinct ranks with non-negligible effect size differences(Tantithamthavorn et al., 2018a; Tantithamthavorn et al., 2016a).”
An empirical study of data sampling techniques for just-in-time software defect prediction
Li, Z., Du, Q., Zhang, H., Jing, X., & Wu, F. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00455-8 · Cited by 21
“To check whether there exist some data sampling algorithms outperform others, we adopt Scott-Knott effect size difference (ESD) test 1 (Tantithamthavorn et al. 2017, 2019) to group and rank all the studied data sampling algorithms.”
Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning
Chen, P., Gong, J., & Chen, T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3525955 · Cited by 36
An empirical study on the effect of community smells on bug prediction
Eken, B., Palma, F., Bener, A., & Tosun, A. (2021) · Software Quality Journal · doi:10.1007/s11219-020-09538-7 · Cited by 16
Evaluating the impact of feature selection consistency in software prediction
Ali, A., & Gravino, C. (2021) · Science of Computer Programming · doi:10.1016/j.scico.2021.102715 · Cited by 16
DP-Share: Privacy-Preserving Software Defect Prediction Model Sharing Through Differential Privacy
Chen, X., Zhang, D., Cui, Z., Gu, Q., & Ju, X. (2019) · Journal of Computer Science and Technology · doi:10.1007/s11390-019-1958-0 · Cited by 18
Preserving Order of Data When Validating Defect Prediction ModelsPreprint
Falessi, D., Narayana, L., Thai, J. F., & Turhan, B. (2018) · arXiv (Cornell University) · Cited by 18
MEG: Multi-objective Ensemble Generation for Software Defect Prediction
Moussa, R., Guizzo, G., & Sarro, F. (2022) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3544902.3546255 · Cited by 17
“These include the use of other diversity measures, and performance measures such as the classification stability [67] and the effort”
BackgroundButchy, A., Jain, U., Leasure, M., Covalesky, V. A., & Mintz, G. S. (2023) · Biomedicines · doi:10.3390/biomedicines11061526 · Cited by 15
“Other methods for verifying model utility exist, such as the “K-fold cross-validation” and the “leave-one-out” meth ds, which provide additional insights regarding th performance of the model [15].”
MethodCross-project defect prediction via semantic and syntactic encoding
Jiang, S., Chen, Y., He, Z., Shang, Y., & Ma, L. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10495-z · Cited by 18
“NumerousrecentresearchworkshaveadoptedtheScott-KnottESD(Tantithamthavorn etal.2016)testtoevaluatethesignificanceoftheirproposedmethods.”
Ma, X., Zou, H., He, P., Keung, J., Li, Y., Yu, X., & Sarro, F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3513413 · Cited by 22
“To determine the statistical significance of the observed performance differences among these data resampling meth-ods, we employ the Scott-Knott Effect Size Difference (ESD) test [44] based on the assumptions of ANalysis Of VAriance (ANOVA).”
Bao, L., Busany, N., Lo, D., & Maoz, S. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00084 · Cited by 19
“[34] proposed the Scott-Knott Effect Size Difference Test, which also accounts for the difference between the means of the groups by incorporating the effect-size measure.”
MethodLittlefield, A. K., Cooke, J. T., Bagge, C. L., Glenn, C. R., Kleiman, E. M., Jacobucci, R., Millner, A. J., & Steinley, D. (2021) · Clinical Psychological Science · doi:10.1177/2167702620961067 · Cited by 16
“However, this validation approach has recently been called into question (see Jacobucci, Littlefield, Millner, Kleiman, & Steinley, 2020; Tantithamthavorn, McIntosh, Hassan, & Matsumoto, 2016).”
Investigating the performance of personalized models for software defect prediction
Eken, B., & Tosun, A. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111038 · Cited by 17
Empirical Validation of cross-version and 10-fold cross-validation for Defect Prediction
Malhotra, R., & Meena, S. (2021) · International Conference on Electronics and Sustainable Communication Systems (ICESC) · doi:10.1109/icesc51422.2021.9533030 · Cited by 18
A Machine Learning Framework for Olive Farms Profit Prediction
Christias, P., & Mocanu, M. (2021) · Water · doi:10.3390/w13233461 · Cited by 19
“6, it can strongly expose the bias-variance trade-off [71].”
BackgroundNiu, J., Li, Z., Chen, H., Dong, X., & Jing, X. (2022) · Software Quality Journal · doi:10.1007/s11219-022-09588-z · Cited by 16
“Recently, Tantithamthavorn et al. (2017) conducted an empirical comparison of 12 model validation strategies from three families (i.e., holdout, cross-validation, and bootstrapping) for within-project defect prediction.”
BackgroundLeveraging the Defects Life Cycle to Label Affected Versions and Defective Classes
Vandehei, B., da Costa, D. A., & Falessi, D. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3433928 · Cited by 18
“lation Coefficient, and Cohen’s Kappa to measure the accuracy for labeling defectiveness in RQ2 and RQ3. Although we do not use the Area Under the Curve (AUC) metric, which is a threshold-free metric [73], our methods do not output probabilities. Therefore, our evaluations are not impacted by threshold choices. 5.4 Ex…”
MethodInfluentialCredibility Based Imbalance Boosting Method for Software Defect Proneness Prediction
Tong, H., Wang, S., & Li, G. (2020) · Applied Sciences · doi:10.3390/app10228059 · Cited by 14
“According to [101], different model validation techniques may affect model performance.”
MethodOn the relationship between design discussions and design quality: a case study of Apache projects
Mannan, U. A., Ahmed, I., Jensen, C., & Sarma, A. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409707 · Cited by 14
“To further analyze the results, we applied Scott-Knott Effect Size Difference (ESD) test [71] to group the difficulty level of finding design discussion in each channel”
BackgroundMethodTempola, F., Rosihan, R., & Adawiyah, R. (2021) · IOP Conference Series Materials Science and Engineering · doi:10.1088/1757-899x/1125/1/012041 · Cited by 14
Does it matter who pays back Technical Debt? An empirical study of self-fixed TD
Tan, J., Feitosa, D., & Avgeriou, P. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106738 · Cited by 16
“As a variant of the Scott–Knott test [39], the Scott–Knott ESD test evaluates a nonnormally distributed dataset and merges any two statistically distinct groups that have a negligible effect size into a single group [38].”
MethodZhu, K., Ying, S., Ding, W., Zhang, N., & Zhu, D. (2021) · Information Sciences · doi:10.1016/j.ins.2021.11.029 · Cited by 14
A Prediction Model for Software Requirements Change Impact
Zamani, K. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678582 · Cited by 14
Multi-source Cross Project Defect Prediction with Joint Wasserstein Distance and Ensemble Learning
Zou, Q., Lu, L., Yang, Z., & Xu, H. (2021) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre52982.2021.00019 · Cited by 14
A comparative study on vectorization methods for non-functional requirements classification
Leelaprute, P., & Amasaki, S. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106991 · Cited by 18
Uncertainty-Aware Robustness Assessment of Industrial Elevator Systems
Han, L., Ali, S., Yue, T., Arrieta, A., & Arratibel, M. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3576041 · Cited by 14
“the Scott-Knott ESD V1 [46] was proposed, which applies log-transformation to correct the nonnormal distribution of the input data.”
MethodPredicting master's applicants performance using KDD techniques
Lopes, R. A., Rodrigues, L., & Brancher, J. D. (2017) · Iberian Conference on Information Systems and Technologies (CISTI) · doi:10.23919/cisti.2017.7975752 · Cited by 13
“The training and test bases were selected by the stratified holdout procedure [10], in the proportion 60/40% respectively.”
MethodTracking bad updates in mobile apps: a search-based approach
Saidani, I., Ouni, A., Ahasanuzzaman, M., Hassan, S., Mkaouer, M. W., & Hassan, A. E. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10125-6 · Cited by 15
“…we measure the features’ importance using the Permutation Feature Importance (PFI) technique (Breiman 2001; Fisher et al. 2019), and then rank them using Scott-Knott (SK) algorithm (Tantithamthavorn et al. 2017; 2018a) in order to prioritize the refactoring efforts during the app’s maintenance.”
MethodInfluentialSelf-citationThe Impact of Duplicate Changes on Just-in-Time Defect Prediction
Duan, R., Xu, H., Fan, Y., & Yan, M. L. (2021) · IEEE Transactions on Reliability · doi:10.1109/tr.2021.3061618 · Cited by 14
“Also, the Scott–Knott effect size difference (SK-ESD) is used to test on the metric importance scores [55].”
MethodST-TLF: Cross-version defect prediction framework based transfer learning
Zhao, Y., Wang, Y., Zhang, Y., Zhang, D., Gong, Y., & Jin, D. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106939 · Cited by 17
“Many studies have pointed out that learning in small samples can produce unstable results [79,80] and proposed Events Per Variable (EPV) to assess the potential impact of the training set, suggesting that the EPV of the training set be greater than 10 [4,81].”
BackgroundMethodInfluentialAn Empirical Study of Refactoring Rhythms and Tactics in the Software Development Process
Noei, S., Li, H., Georgiou, S., & Zou, Y. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3326775 · Cited by 24
“Using the Scott-Knott-ESD [31], [32], we group combinations of refactoring rhythms and project or author profiles into statistically significant clusters.”
MethodInfluentialMalhotra, R., Chawla, S., & Sharma, A. (2025) · Neural Computing and Applications · doi:10.1007/s00521-024-10902-y · Cited by 21
On cost-effective software defect prediction: Classification or ranking?
Huo, X., & Li, M. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.05.100 · Cited by 13
A Software Defect Learning and Analysis Utilizing Regression Method for Quality Software Development
Jadhav, R. (2019) · International Journal of Advanced Trends in Computer Science and Engineering · doi:10.30534/ijatcse/2019/38842019 · Cited by 13
Imbalanced metric learning for crashing fault residence prediction
Zhou, X., Zhao, K., Yan, M. L., Yuan, P., Xu, L., Lei, Y., & Zhang, X. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110763 · Cited by 16
“4 visualizes the corresponding statistical test re-sults by SKESD for our IML method and the nine sampling based imbalanced learning methods.”
MethodInfluentialAn instance-oriented performance measure for classification
Yu, S., Li, X., Feng, Y., Zhang, X., & Chen, S. (2021) · Information Sciences · doi:10.1016/j.ins.2021.08.094 · Cited by 13
A cross‐project defect prediction method based on multi‐adaptation and nuclear norm
Huang, Q., Ma, L., Jiang, S., Wu, G., Song, H., Jiang, L., & Zheng, C. (2021) · IET Software · doi:10.1049/sfw2.12053 · Cited by 13
An empirical study on the issue reports with questions raised during the issue resolving process
Huang, Y., da Costa, D. A., Zhang, F., & Zou, Y. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9636-3 · Cited by 13
Guo, S., Dong, J., Li, H., & Wang, J. (2021) · Journal of Software Evolution and Process · doi:10.1002/smr.2362 · Cited by 14
“30 There are also many studies focusing on software defect prediction, 31 – 34 which mainly apply machine learning to two parts; one is Within-Project Defect Prediction (WPDP), and the other is Cross-Project Defect Prediction (CPDP).”
BackgroundPairing Machine Learning and Clinical Psychology: How You Evaluate Predictive Performance Matters
Jacobucci, R., Littlefield, A. K., Millner, A. J., Kleiman, E., & Steinley, D. (2020) · doi:10.31234/osf.io/2yber · Cited by 12
Feature sets in just-in-time defect prediction: an empirical evaluation
Bludau, P., & Pretschner, A. (2022) · doi:10.1145/3558489.3559068 · Cited by 12
“We execute the Non-Parametric ScottKnott ESD (NPSK) test [38] to determine which features are the most important ones in the reduced set, similar to previous studies [13, 41, 42].”
MethodExploring better alternatives to size metrics for explainable software defect prediction
Chai, C., Fan, G., Yu, H., Huang, Z., Ding, J., & Guan, Y. (2023) · Software Quality Journal · doi:10.1007/s11219-023-09656-y · Cited by 15
“To verify whether the experimental results are statistically significant, we exploit SK-ESD (Scott-Knott effect size difference) (Tantithamthavorn et al., 2017) test, which is based on hierarchical cluster analysis and divides the results into statistically distinct groups.”
High‐frequency multi‐solute calibration using an in situ UV –visible sensor
Pesántez, J., Birkel, C., Mosquera, G. M., Peña, P., Arízaga‐Idrovo, V., Mora, E., McDowell, W. H., & Crespo, P. (2021) · Hydrological Processes · doi:10.1002/hyp.14357 · Cited by 13
“These data sets were divided into six subsets and we randomly selected a 30% sample of each of them to compose the validation subset (Arsenault et al., 2018; Fernandez-Palomino et al., 2020; Tantithamthavorn et al., 2017).”
MethodEnhanced Churn Prediction Model with Boosted Trees Algorithms in The Banking Sector
Sagala, N. T. M., & Permai, S. D. (2021) · International Conference on Data Science and Its Applications (ICoDSA) · doi:10.1109/icodsa53588.2021.9617503 · Cited by 12
What happens in my code reviews? An investigation on automatically classifying review changes
Fregnan, E., Petrulio, F., Geronimo, L. D., & Bacchelli, A. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10075-5 · Cited by 19
“In addition, the validation strategy may be object of discussion: according to a recent paper (Tantithamthavorn et al. 2016), the 10-fold cross-validation may bias the interpretation of the results, as it relies on random splitting of the data used to train and test the model.”
BackgroundMethodInfluentialThe impact of class imbalance techniques on crashing fault residence prediction models
Zhao, K., Zhou, X., Yan, M. L., Zhang, T., Xue, L., Fan, M. Y., & Keung, J. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10294-y · Cited by 13
“To investigate the significant difference of the experimental results for crashing faults residence prediction task, we employ the advanced S cott-K nott E ffect S ize D ifference ( SKESD ) test proposed by Tantithamthavorn et al. (2016).”
MethodA Survey on Deep Learning-Based Source Code Defect Analysis
Guan, Z., Wang, X., Wei, X., Wang, J., & Zhang, L. (2020) · International Conference on Computer and Communication Systems (ICCCS) · doi:10.1109/icccs49078.2020.9118556 · Cited by 12
“In the earlier methods, researches regard the source code as a combination of token fragmentation, and match them based on information retrieval to achieve SCDA, including clone detection [3]-[6], vulnerability prediction [7] [8], bug location [9], etc.”
MethodConditional Domain Adversarial Adaptation for Heterogeneous Defect Prediction
Gong, L., Jiang, S., & Jiang, L. (2020) · IEEE Access · doi:10.1109/access.2020.3017101 · Cited by 13
“GANs transfer the source instances into target instance space, and then use the transferred source instances to train RF classifier [46], [47].”
MethodPecorelli, F., & Nucci, D. D. (2021) · Science of Computer Programming · doi:10.1016/j.scico.2021.102611 · Cited by 13
“We used this validation strategy since it is among the least biased and most stable validation approaches, according to the findings reported by Tantithamthavorn et al. [86].”
BackgroundMethodResultInfluentialDo practitioners intentionally self-fix Technical Debt and why?
Tan, J., Feitosa, D., & Avgeriou, P. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00029 · Cited by 15
Yet Another Model! A Study on Model’s Similarities for Defect and Code Smells
dos Santos, G. E., Santana, A., Vale, G., & Figueiredo, E. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-30826-0_16 · Cited by 16
“…Random Forest [24], Decision Tree [17], Extra Trees [6], Logistic Regression [30], K-Neighbors Classifier (KNN) [76], Gradient Boosting Machine [73], Extreme Gradient Boosting [64], Linear Discriminant Analysis [6], Ada Boost Classifier [56], Light Gradient Boosting Machine [33], Naive Bayes…”
MethodLocal and Global Explainability for Technical Debt Identification
Tsoukalas, D., Mittas, N., Arvanitou, E.-M., Ampatzoglou, A., Chatzigeorgiou, A., & Kehagias, D. D. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3422427 · Cited by 12
“More speci fi cally, we made use of a variant of the original approach, namely the Scott-Knott Effect Size Difference (SK-ESD) test [51] that takes into consideration the effect size of an observed difference that is related to the practical importance of the derived fi ndings in the examined…”
MethodAutomatic software tailoring for Green Internet of Things
Aragón-Jurado, J. M., de la Torre, J. C., Ruiz, P., & Dorronsoro, B. (2025) · Internet of Things · doi:10.1016/j.iot.2025.101521 · Cited by 16
A Survey on Botnets Attack Detection Utilizing Machine and Deep Learning Models
Alomari, D. M., Anis, F. M., Alabdullatif, M., & Aljamaan, H. (2023) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3593434.3593967 · Cited by 14
“Unreliable Model Validation : Most of the proposed ML and DL models were evaluated using the Hold-Out method, which produces biased with high variance experimentation outcomes [20].”
MethodRevisiting ‘revisiting supervised methods for effort‐aware cross‐project defect prediction’
Li, F., Yang, P., Keung, J., Hu, W., Luo, H., & Yu, X. (2023) · IET Software · doi:10.1049/sfw2.12133 · Cited by 14
“Then, we use the Scott ‐ Knott Effect Size Difference (ESD) [39] test to group these EACPDP methods into different rankings.”
MethodInfluentialZhu, K., Zhang, N., Jiang, C., & Zhu, D. (2023) · Software Practice and Experience · doi:10.1002/spe.3274 · Cited by 11
Metadata
- Authors
- Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K.
- Affiliations
- Nara Institute of Science and Technology (JP)
- Queen's University (CA)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2017-01-01
- Volume
- 43
- Issue
- 1
- Pages
- 1-18
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2474835145
- Cited by
- 574
- References
- 129
- Citations by year
- 2026: 39 · 2025: 53 · 2024: 69 · 2023: 56 · 2022: 58 · 2021: 93 · 2020: 63 · 2019: 75 · 2018: 45 · 2017: 18
- FWCI
- 106.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Funders
- JSPS
- Advancing Strategic International Networks to Accelerate the Circulation of Talented Researchers
- Interdisciplinary Global Networks for Accelerating Theory and Practice in Software Ecosystem
- JSPS Fellows (16J03360)
- Natural Sciences and Engineering Research Council of Canada (NSERC)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- fault prediction, model validation, k-fold cross-validation, software quality assurance, defect dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.