An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
The DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index.
Abstract
Defect prediction models help software quality assurance teams to allocate their limited resources to the most defect-prone modules. Model validation techniques, such as$k$-fold cross-validation, use historical data to estimate how well a model will perform in the future. However, little is known about how accurate the estimates of model validation techniques tend to be. In this paper, we investigate the bias and variance of model validation techniques in the domain of defect prediction. Analysis of 101 public defect datasets suggests that 77 percent of them are highly susceptible to producing unstable results– - selecting an appropriate model validation technique is a critical experimental design choice. Based on an analysis of 256 studies in the defect prediction literature, we select the 12 most commonly adopted model validation techniques for evaluation. Through a case study of 18 systems, we find that single-repetition holdout validation tends to produce estimates with 46-229 percent more bias and 53-863 percent more variance than the top-ranked model validation techniques. On the other hand, out-of-sample bootstrap validation yields the best balance between the bias and variance of estimates in the context of our study. Therefore, we recommend that future defect prediction studies avoid single-repetition holdout validation, and instead, use out-of-sample bootstrap validation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Biases may be balanced with variance of performance estimates using out-of-sample bootstrap techniques [139].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [139]
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
An Analysis of Transformations
Box, G. E. P., & Cox, D. R. (1964) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1964.tb00553.x · Cited by 15,052
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Research works printed but not linked: 9
- 10.29172/7c2a6982-6d72-4cd8-bba6-2fccb06a7011 · DOI · Paper page
- 10.2307/2289236 · doi:10.2307/2289236 · Paper page
- Jelihovschi, 2014, The ScottKnott Clustering Algorithm
- 10.1067/mod.2000.109032 · DOI · Paper page
- Meyer, 2014, E1071: Misc functions of the department of statistics (E1071), TU Wien
- 10.1109/ICSE.2005.1553571 · DOI · Paper page
- 2013, R Core Team, R: A language and environment for statistical computing, R foundation for statistical computing, Vienna, Austria
- 10.1007/978-3-031-01899-2 · doi:10.1007/978-3-031-01899-2 · Paper page
- Tantithamthavorn, 2015, ScottKnottESD: An R package of the Scott-Knott effect size difference (ESD) test
Cited by
A Novel Neural Source Code Representation Based on Abstract Syntax Tree
Zhang, J., Wang, X., Zhang, H., Sun, H., Wang, K., & Liu, X. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00086 · Cited by 655
“Many software engineering methods, such as source code classification [1], [2], code clone detection [3], [4], [5], [6], defect prediction [7], [8] and code summarization [9], [10] have been proposed to improve software development and maintenance.”
MethodThe Impact of Automated Parameter Optimization on Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2794977 · Cited by 415
“Our recent work shows that defect prediction models that are trained using datasets with low EPV values are especially susceptible to unstable results [141].”
BackgroundMethodInfluentialSelf-citationAutomated parameter optimization of classification techniques for defect prediction models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884857 · Cited by 331
Self-citationA framework for data-driven digital twins of smart manufacturing systems
Friederich, J., Francis, D. P., Lazarova-Molnar, S., & Mohamed, N. (2022) · Computers in Industry · doi:10.1016/j.compind.2021.103586 · Cited by 323
“It has been proposed that out-of-sample bootstrap techniques tend to give estimates that are the least biased (Tantithamthavorn et al., 2017).”
BackgroundTantithamthavorn, C., Hassan, A. E., & Matsumoto, K. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876537 · Cited by 327
Self-citationDeep Semantic Feature Learning for Software Defect Prediction
Wang, S., Liu, T., Nam, J., & Tan, L. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2877612 · Cited by 320
Detecting code smells using machine learning techniques: Are we there yet?
Di Nucci, D., Palomba, F., Tamburri, D. A., Serebrenik, A., & De Lucia, A. (2018) · 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2018.8330266 · Cited by 305
What’s in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform
Borges, H., & Tulio Valente, M. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.09.016 · Cited by 317
“, Scott-Knott ESD [64]) may provide additional insights on the factors that impact the classification of a project in a given growth pattern.”
BackgroundAzeem, M. I., Palomba, F., Shi, L., & Wang, Q. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.12.009 · Cited by 289
“[60] demonstrated that this validation strategy is among the most biased and less reliable.”
BackgroundMethodHeterogeneous Defect Prediction
Nam, J., Fu, W., Kim, S., Menzies, T., & Tan, L. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2720603 · Cited by 277
Progress on approaches to software defect prediction
Li, Z., Jing, X.-Y., & Zhu, X. (2018) · IET Software · doi:10.1049/iet-sen.2017.0148 · Cited by 264
“To examine the bias and variance of model validation techniques in the field of defect prediction, Tantithamthavorn et al. [70] conducted an in-depth study with 12 most widely used model validation techniques for the evaluation.”
MethodVulRepair: a T5-based automated software vulnerability repair
Fu, M., Tantithamthavorn, C., Le, T., Nguyen, V., & Phung, D. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549098 · Cited by 207
Self-citationWhat do developers search for on the web?
Xia, X., Bao, L., Lo, D., Kochhar, P. S., Hassan, A. E., & Xing, Z. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9514-4 · Cited by 174
“’s study (Tantithamthavorn et al. 2017), the Scott-Knott test assumed that the data is normally distributed, and it might create groups that are trivially different from one another.”
MethodInfluentialSelf-citationPerceptions, Expectations, and Challenges in Defect Prediction
Wan, Z., Xia, X., Hassan, A. E., Lo, D., Yin, J., & Yang, X. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2877678 · Cited by 169
“…7 categories (followed by identified papers): Prediction models and learning approaches ([14], [20], [21], [32], [35], [52], [74], [75], [83], [85]) Metrics in defect prediction models ([8], [13], [25], [48], [77], [89], [95], [97], [98], [101]) Cross-project/cross-company defect prediction…”
MethodSelf-citationOperationalising ethics in artificial intelligence for healthcare: a framework for AI developers
Solanki, P., Grundy, J., & Hussain, W. (2023) · AI and Ethics · doi:10.1007/s43681-022-00195-z · Cited by 157
“Avoid unethical corrections to models that sacrifice fairness for accuracy [68, 139]”
BackgroundAn Empirical Study of Model-Agnostic Techniques for Defect Prediction Models
Jiarpakdee, J., Tantithamthavorn, C. K., Dam, H. K., & Grundy, J. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2982385 · Cited by 148
“We use the out-of-sample bootstrap validation technique (1) to ensure that the generated training samples are representative to the original dataset and (2) to ensure that the produced estimates are the least bias and most reliable [101].”
MethodInfluentialSelf-citationMULTI: Multi-objective effort-aware just-in-time software defect prediction
Chen, X., Zhao, Y., Wang, Q., & Yuan, Z. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.08.004 · Cited by 156
Categorizing the Content of GitHub README Files
Prana, G. A. A., Treude, C., Thung, F., Atapattu, T., & Lo, D. (2019) · Empirical Software Engineering · doi:10.1007/s10664-018-9660-3 · Cited by 143
“therwise it is a negative instance. We learn an SVM classier to get the weight of each feature. To capture signicantly important features, we perform the Scott-Knott ESD (Eect Size Dierence) test [69]. For the purpose of this test, we perform ten times ten-fold cross validation where each cross validation generates di…”
MethodAccessibility issues in Android apps: state of affairs, sentiments, and ways forward
Alshayban, A., Ahmed, I., & Malek, S. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380392 · Cited by 142
“[45] proposed ESD as it does not require the data to be normally distributed.”
BackgroundMethodInvestigation on the stability of SMOTE-based oversampling techniques in software defect prediction
Feng, S., Keung, J., Yu, X., Xiao, Y., & Zhang, M. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106662 · Cited by 173
Predicting Defective Lines Using a Model-Agnostic Technique
Wattanakriengkrai, S., Thongtanunam, P., Tantithamthavorn, C., Hata, H., & Matsumoto, K. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3023177 · Cited by 137
Self-citationMining Software Defects: Should We Consider Affected Releases?
Yatish, S., Jiarpakdee, J., Thongtanunam, P., & Tantithamthavorn, C. K. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00075 · Cited by 156
“[57, 64] demonstrated that it produces the most stable performance estimates when comparing to other model validation techniques like k-fold cross-validation.”
MethodSelf-citationNot all bugs are the same: Understanding, characterizing, and classifying bug types
Catolino, G., Palomba, F., Zaidman, A., & Ferrucci, F. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.03.002 · Cited by 179
“While such a strategy has recently been criticized [79], we tackled its”
BackgroundRidge and Lasso Regression Models for Cross-Version Defect Prediction
Yang, X., & Wen, W. (2018) · IEEE Transactions on Reliability · doi:10.1109/tr.2018.2847353 · Cited by 149
“[19] investigated the bias and variance of 12 model validation techniques in the domain of defect prediction, and they pointed out that selecting an appropriate model validation technique was a critical experimental design choice.”
BackgroundSupervised vs Unsupervised Models: A Holistic Look at Effort-Aware Just-in-Time Defect Prediction
Huang, Q., Xia, X., & Lo, D. (2017) · 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme.2017.51 · Cited by 124
“, choosing other time slots) may produce different performance estimates [53].”
BackgroundMalhotra, R., & Kamal, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.04.090 · Cited by 150
Balogun, A. O., Basri, S. B., Abdulkadir, S. J., & Hashim, A. S. (2019) · Applied Sciences · doi:10.3390/app9132764 · Cited by 150
The Impact of Feature Importance Methods on the Interpretation of Defect Classifiers
Rajbahadur, G. K., Wang, S., Oliva, G. A., Kamei, Y., & Hassan, A. E. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3056941 · Cited by 181
“[18, 77, 78] investigate how various experimental design choices impact the computed feature importance ranks of a classifier.”
BackgroundMethodSelf-citationDeepLineDP: Towards a Deep Learning Approach for Line-Level Defect Prediction
Pornprasit, C., & Tantithamthavorn, C. K. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3144348 · Cited by 165
“%) as follows: Finally, we apply a ScottKnott ESD test to cluster the distributions into statistically distinct ranks with non-negligible effect size difference [59], [60], [61].”
MethodSelf-citationFeng, S., Keung, J., Yu, X., Xiao, Y., Bennin, K. E., Kabir, M. A., & Zhang, M. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106432 · Cited by 152
“These datasets were selected from the PROMISE repository and have been used in several previous studies [12,65,75].”
MethodImproving defect prediction with deep forest
Zhou, T., Sun, X., Xia, X., Li, B., & Chen, X. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.07.003 · Cited by 139
An Improved CNN Model for Within-Project Software Defect Prediction
Pan, C., Lu, M., Xu, B., & Gao, H. (2019) · Applied Sciences · doi:10.3390/app9102138 · Cited by 138
“As suggested by Arcuri [27], to minimize statistical bias and variance, there should be no fewer than 30 repeated experiments, and research on model validation by Tantithamthavorn [28] also supported that conclusion.”
ResultZhu, K., Ying, S., Zhang, N., & Zhu, D. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111026 · Cited by 141
An experience report on defect modelling in practice: pitfalls and challenges
Tantithamthavorn, C., & Hassan, A. E. (2018) · Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice · doi:10.1145/3183519.3183547 · Cited by 122
“Model validation techniques (e.g., out-of-samplebootstrapvalidation,repeated k -foldcross-validation) are commonly used to estimate the model performance on an un-seen part of the dataset [76].”
BackgroundMethodInfluentialSelf-citationToward a Smell-Aware Bug Prediction Model
Palomba, F., Zanoni, M., Fontana, F. A., Lucia, A. D., & Oliveto, R. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2770122 · Cited by 124
“the likelihood of a model to be in the top Scott-Knott ESD rank, as done in previous work [90], [118], [119].”
BackgroundMethodResultInfluentialThe Impact of Mislabeled Changes by SZZ on Just-in-Time Defect Prediction
Fan, Y., Xia, X., da Costa, D. A., Lo, D., Hassan, A. E., & Li, S. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2929761 · Cited by 144
“Next, following prior studies [19], [73], [79], we apply the Scott-Knott Effect Size Difference (SK-ESD) test [73] on the metric importance scores from the 1,000 iterations of bootstrap.”
InfluentialSelf-citationThe impact of feature reduction techniques on defect prediction models
Kondo, M., Bezemer, C., Kamei, Y., Hassan, A. E., & Mizuno, O. (2019) · Empirical Software Engineering · doi:10.1007/s10664-018-9679-5 · Cited by 115
Self-citationUnderstanding machine learning software defect predictions
dos Santos, G. E., Figueiredo, E., Veloso, A., Viggiato, M., & Ziviani, N. (2020) · Automated Software Engineering · doi:10.1007/s10515-020-00277-4 · Cited by 130
“The results reported are the average of the ten runs, and to ensure their relevance we assess the statistical significance of our measurements using the Scott-Knott Effect Difference test Tantithamthavorn et al. (2017, 2018).”
MethodOn the Multiple Sources and Privacy Preservation Issues for Heterogeneous Defect Prediction
Li, Z., Jing, X., Zhu, X., Zhang, H., Xu, B., & Ying, S. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2780222 · Cited by 106
“Prior works [6], [84], [85] point out that Nemenyi’s posthoc test produces overlapping groups of classification techniques.”
BackgroundMethodInfluentialXu, J., Wang, F., & Ai, J. (2020) · IEEE Transactions on Reliability · doi:10.1109/tr.2020.3040191 · Cited by 114
“[26],[27].”
Keeping Deep Learning Models in Check: A History-Based Approach to Mitigate Overfitting
Li, H., Rajbahadur, G. K., Lin, D., Bezemer, C., & Jiang, Z. M. (2024) · IEEE Access · doi:10.1109/access.2024.3402543 · Cited by 129
“Tantithamthavorn et al. [72] evaluated 12 model validation techniques specifically for defect prediction models and concluded that out-of-sample bootstrap emerges as the least biased and most stable technique that helps detect overfit.”
BackgroundIba, K., Shinozaki, T., Maruo, K., & Noma, H. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01201-w · Cited by 123
How Practitioners Perceive Automated Bug Report Management Techniques
Zou, W., Lo, D., Chen, Z., Xia, X., Feng, Y., & Xu, B. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2870414 · Cited by 96
“To address the limitations of Scott-Knott test, Tantithamthavorn et al. proposed the Scott-Knott Effect Size Difference (ESD) test to correct the non-normal distribution of an input dataset and merge any two statistically distinct groups whose difference has a negligible effect size into one group [73].”
BackgroundMethodInfluentialChaff from the Wheat: Characterizing and Determining Valid Bug Reports
Fan, Y., Xia, X., Lo, D., & Hassan, A. E. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2864217 · Cited by 114
“To determine which of the features are the most important, we apply the Scott-Knott Effect Size Difference (ESD) test [51], [81], [93] for the importance values taken from all 10 runs of 10-fold cross-validation.”
MethodSelf-citationClass Imbalance Evolution and Verification Latency in Just-in-Time Software Defect Prediction
Cabral, G. G., Minku, L. L., Shihab, E., & Mujahid, S. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00076 · Cited by 114
“[33], [34], even though they used Cohen’s parametric effect size.”
MethodToo Many User-Reviews! What Should App Developers Look at First?
Noei, E., Zhang, F., & Zou, Y. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2893171 · Cited by 93
“criteria: (i) the apps that get updated less than ten times during our study as we study the relation of addressing the issues raised by the key topicswith star-ratings over time, and (ii) the apps that receive less than ten user-reviews between each update to avoid the apps with few user-reviews skew the findings [17…”
BackgroundMachine/Deep Learning for Software Engineering: A Systematic Literature Review
Wang, S., Huang, L., Gao, A., Ge, J., Zhang, T., Feng, H., Satyarth, I., Li, M., Zhang, H., & Ng, V. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3173346 · Cited by 89
“Many SLRs, surveys and comparative studies [183], [394], [395], [396], [397], [398], [399], [400] have focused on investigating the use of ML/DL in software defect prediction.”
BackgroundYang, Z., Wei, T., Liang, Y., Yuan, X., Gao, R., Xia, Y., Zhou, J., Zhang, Y., & Yu, Z. (2025) · Nature Communications · doi:10.1038/s41467-025-57587-y · Cited by 148
“The dataset is available at https://web.inf.ufpr.br/vri/databases/breast-cancer-histopathological-database-breakhis/.”
An empirical study on pareto based multi-objective feature selection for software defect prediction
Ni, C., Chen, X., Wu, F., Shen, Y., & Gu, Q. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.03.012 · Cited by 96
A Comparative Study of Class Rebalancing Methods for Security Bug Report Classification
Zheng, W., Xun, Y., Wu, X., Deng, Z., Chen, X., & Sui, Y. (2021) · IEEE Transactions on Reliability · doi:10.1109/tr.2021.3118026 · Cited by 85
Ma, X., Xu, J., Wu, P., & Peng, K. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3126175 · Cited by 115
“In this study, we adopted the holdout validation training method [63] for the models, i.e., the training set and the validation set were input into the models simultaneously.”
MethodKusuma, K., Larsen, M. E., Quiroz, J. C., Gillies, M. B., Burnett, A., Qian, J., & Torok, M. H. (2022) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2022.09.050 · Cited by 101
The Impact of Correlated Metrics on the Interpretation of Defect Models
Jiarpakdee, J., Tantithamthavorn, C. K., & Hassan, A. E. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2891758 · Cited by 146
“clusions are statistically sound and robust, we use the outof-sample bootstrap validation technique, which leverages aspects of statistical inference [17], [19], [29], [72], [78].”
BackgroundMethodSelf-citationPredicting Node Failures in an Ultra-Large-Scale Cloud Computing Platform: An AIOps Solution
Li, Y., Jiang, Z. M. (., Li, H., Hassan, A. E., He, C., Huang, R., Zeng, Z., Wang, M., & Chen, P. (2020) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3385187 · Cited by 82
Self-citationCode smell detection using feature selection and stacking ensemble: An empirical investigation
Alazba, A., & Aljamaan, H. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106648 · Cited by 101
The impact of automated feature selection techniques on the interpretation of defect models
Jiarpakdee, J., Tantithamthavorn, C. K., & Treude, C. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09848-1 · Cited by 95
Self-citationGallaba, K., & McIntosh, S. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2838131 · Cited by 114
“Finally, we apply the Scott-Knott Effect Size Difference (ESD) test [31]—an enhancement to the Scott-Knott test [27], which also considers the effect size when clustering CI sections into statistically distinct ranks.”
MethodSelf-citationExplainable AI for Software Engineering
Tantithamthavorn, C. K., & Jiarpakdee, J. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678580 · Cited by 102
“• Use the Scott-Knott ESD test to identify the best classification techniques that are statistically significantly different with non-negligible Cliff’s |δ| effect size [14].”
MethodSelf-citationAutomating Change-Level Self-Admitted Technical Debt Determination
Yan, M. L., Xia, X., Shihab, E., Lo, D., Yin, J., & Yang, X. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2831232 · Cited by 89
“To determine which of the features are the most important, we apply Scott-Knott Effect Size Difference (ESD) test for the importance values taken from all 10 runs of 10-fold cross-validation [64], [66], [67].”
MethodTowards just-in-time suggestions for log changes
Li, H., Shang, W., Zou, Y., & Hassan, A. E. (2016) · Empirical Software Engineering · doi:10.1007/s10664-016-9467-z · Cited by 103
“In this work, we use an enhanced SK approach [36], which considers the effect size in addition to the statistical significance of the difference between groups, to divide the factors into distinct groups according to their influence in the random forest classifiers.”
MethodSelf-citationSoftware defect prediction model based on LASSO–SVM
Wang, K., Liu, L., Yuan, C., & Wang, Z. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-04960-1 · Cited by 77
On the Impact of Sample Duplication in Machine-Learning-Based Android Malware Detection
Zhao, Y., Li, L., Wang, H., Cai, H., Bissyandé, T. F., Klein, J., & Grundy, J. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3446905 · Cited by 74
“Although we have selected four well-known algorithms and both in-the-lab and in-the-wild experimental settings, which have also been frequently leveraged by other researchers to achieve similar purposes, they may not be entirely suitable for predicting Android malware [59].”
MethodImproving the prediction of continuous integration build failures using deep learning
Saidani, I., Ouni, A., & Mkaouer, M. W. (2022) · Automated Software Engineering · doi:10.1007/s10515-021-00319-5 · Cited by 94
Balogun, A. O., Basri, S. B., Mahamad, S., Abdulkadir, S. J., Almomani, M. A., Adeyemo, V. E., Al-Tashi, Q., Mojeed, H. A., Imam, A. A., & Bajeh, A. O. (2020) · Symmetry · doi:10.3390/sym12071147 · Cited by 82
“The Scott–KnottESD rank test has been suggested and widely used in existing SDP studies [2,5,25,28,54].”
Ni, W., Wang, T., Wu, Y., Liu, X., Li, Z., Yang, R., Zhang, K., Yang, J., Zeng, M., Hu, N., et al. (2024) · Sensors and Actuators B Chemical · doi:10.1016/j.snb.2024.136206 · Cited by 77
Cross Project Defect Prediction via Balanced Distribution Adaptation Based Transfer Learning
Xu, Z., Pang, S., Zhang, T., Luo, X., Liu, J., Tang, Y., Yu, X., & Xue, L. (2019) · Journal of Computer Science and Technology · doi:10.1007/s11390-019-1959-z · Cited by 78
“Unlike the ANOVA test [60] and Scott-Knott test [30 , 61] which are based on an assumption that the performance values should fulfill the normality of both the residuals assumption and the ho-moscedasticity assumption [62] , the statistical tests used in this work are non-parametric, which means…”
MethodPractitioners’ Perceptions of the Goals and Visual Explanations of Defect Prediction Models
Jiarpakdee, J., Tantithamthavorn, C. K., & Grundy, J. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00055 · Cited by 100
Self-citationCross-version defect prediction via hybrid active learning with kernel principal component analysis
Xu, Z., Liu, J., Luo, X., & Zhang, T. (2018) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2018.8330210 · Cited by 81
“Recent studies have examined the impact of different classification models [21], [22], feature selection methods [15], sampling approaches [23], and model validation techniques [24] on the performance of WPDP.”
BackgroundLDFR: Learning deep feature representation for software defect prediction
Xu, Z., Li, S., Xu, J., Liu, J., Luo, X., Zhang, Y., Zhang, T., Keung, J., & Tang, Y. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.110402 · Cited by 75
Croft, R., Newlands, D., Chen, Z., & Babar, M. A. (2021) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3475716.3475781 · Cited by 67
“[57] to use out-of-sample bootstrap for evaluation.”
BackgroundGraph Neural Network for Source Code Defect Prediction
Šikić, L., Kurdija, A. S., Vladimir, K., & Šilić, M. (2022) · IEEE Access · doi:10.1109/access.2022.3144598 · Cited by 76
“However, as suggested in previous statistical studies [46], [47], [48], an experiment should be repeated at least 30 times to minimize the statistical bias and variance of the results; otherwise, the risk of obtaining non-repeatable results and drawing wrong conclusions increases [49].”
The Impact of Using Regression Models to Build Defect Classifiers
Rajbahadur, G. K., Wang, S., Kamei, Y., & Hassan, A. E. (2017) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2017.4 · Cited by 80
“SK-ESD uses the effect size as computed by Cohen’s ∆ [12] to merge statistically similar groups into the same rank.”
BackgroundMethodInfluentialSelf-citationIntegrated Approach to Software Defect Prediction
Felix, E. A., & Lee, S. P. (2017) · IEEE Access · doi:10.1109/access.2017.2759180 · Cited by 79
“a model will perform on unseen data [61].”
MethodAutoSpearman: Automatically Mitigating Correlated Software Metrics for Interpreting Defect Models
Jiarpakdee, J., Tantithamthavorn, C., & Treude, C. (2018) · 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme.2018.00018 · Cited by 65
Self-citationDeveloper-Driven Code Smell Prioritization
Pecorelli, F., Palomba, F., Khomh, F., & Lucia, A. D. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387457 · Cited by 76
“because the randomness with which it creates training and test data may lead to an under- or over-estimation of the real performance of a machine learning model—this is especially true in the case of classification algorithms [87].”
BackgroundMethodInfluentialCross-Project Just-in-Time Bug Prediction for Mobile Apps: An Empirical Assessment
Catolino, G., Nucci, D. D., & Ferrucci, F. (2019) · IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft) · doi:10.1109/mobilesoft.2019.00023 · Cited by 71
“We statistically verified the validity of our findings by using the Scott-Knott ESD test [80].”
MethodResultSoftware defect prediction based on correlation weighted class association rule mining
Shao, Y., Liu, B., Wang, S., & Li, G. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.105742 · Cited by 66
Heterogeneous defect prediction with two-stage ensemble learning
Li, Z., Jing, X., Zhu, X., Zhang, H., Xu, B., & Ying, S. (2019) · Automated Software Engineering · doi:10.1007/s10515-019-00259-1 · Cited by 70
An Empirical Study on Heterogeneous Defect Prediction Approaches
Chen, H., Jing, X., Li, Z., Wu, D., Yi, P., & Huang, Z. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2968520 · Cited by 72
PyExplainer: Explaining the Predictions of Just-In-Time Defect Models
Pornprasit, C., Tantithamthavorn, C. K., Jiarpakdee, J., Fu, M. C., & Thongtanunam, P. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678763 · Cited by 93
“To ensure that the evaluation of our just-in-time defect prediction reflects a real-world scenario, we first sort the date of the commits to preserve the order of the commits in a chronological order [30, 43].”
BackgroundSelf-citationAn empirical study of security warnings from static application security testing tools
Aloraini, B., Nagappan, M., Germán, D. M., Hayashi, S., & Higo, Y. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.110427 · Cited by 73
An Empirical Study of the Impact of Data Splitting Decisions on the Performance of AIOps Solutions
Lyu, Y., Li, H., Sayagh, M., Jiang, Z. M. (., & Hassan, A. E. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3447876 · Cited by 60
“, the baseline model in our context) after the model evaluation [82].”
BackgroundMethodInfluentialSelf-citationCommentFinder: a simpler, faster, more accurate code review comments recommendation
Hong, Y., Tantithamthavorn, C., Thongtanunam, P., & Aleti, A. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549119 · Cited by 57
Self-citationThe impact of context metrics on just-in-time defect prediction
Kondo, M., Germán, D. M., Mizuno, O., & Choi, E.-H. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09736-3 · Cited by 69
Explainable AI for Android Malware Detection: Towards Understanding Why the Models Perform So Well?
Liu, Y., Tantithamthavorn, C. K., Li, L., & Liu, Y. (2022) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre55969.2022.00026 · Cited by 71
“Prior studies [45], [46], [47], [10] raise concerns that the experimental components often have a large impact on the accuracy and explanations of defect prediction models (e.g., data quality [48], class imbalance [49], parameter settings [50], model validation techniques [51]).”
BackgroundMethodSelf-citationA comprehensive comparative study of clustering-based unsupervised defect prediction models
Xu, Z., Li, L., Yan, M. L., Liu, J., Luo, X., Grundy, J., Zhang, Y., & Zhang, X. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110862 · Cited by 59
Abdu, A., Zhai, Z., Abdo, H. A., Algabri, R., Al‐masni, M. A., Muhammad, M. S., & Gu, Y. H. (2024) · Scientific Reports · doi:10.1038/s41598-024-65639-4 · Cited by 54
“According to our findings in previous studies 38,64 , validation techniques like k-fold cross-validation often introduce significant bias when evaluating SDP models, leading to inaccurate assessments.”
ResultImproving change prediction models with code smell-related information
Catolino, G., Palomba, F., Fontana, F. A., Lucia, A. D., Zaidman, A., & Ferrucci, F. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09739-0 · Cited by 62
“size|measured using Cli’s Delta (or d) [39]|is negligible, so that the creation of trivial groups is avoided. To perform the test, we relied on the implementation provided by Tantithamthavorn et al. [114]. 3.4 RQ 2 - Comparison between Intensity Index and Antipattern Metrics In RQ 2 our goal was to compare the perform…”
InfluentialSource code properties of defective infrastructure as code scripts
Rahman, A., & Williams, L. A. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.04.013 · Cited by 72
A large-scale empirical study on vulnerability distribution within projects and the lessons learned
Liu, B., Meng, G., Zou, W., Gong, Q., Li, F., Lin, M., Sun, D., Huo, W., & Zhang, C. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380923 · Cited by 54
“These insights are further utilized to detect more related bugs [9, 53, 54].”
BackgroundImplications of deep learning for the automation of design patterns organization
Hussain, S., Keung, J., Khan, A., Ahmad, A., Cuomo, S., Piccialli, F., Jeon, G., & Akhunzada, A. (2017) · J. Parallel Distributed Comput. · doi:10.1016/j.jpdc.2017.06.022 · Cited by 52
The impact of human factors on the participation decision of reviewers in modern code review
Ruangwan, S., Thongtanunam, P., Ihara, A., & Matsumoto, K. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9646-1 · Cited by 72
Self-citationAn exploratory study on confusion in code reviews
Ebert, F., Castor, F., Novielli, N., & Serebrenik, A. (2021) · Empirical Software Engineering · doi:10.1007/s10664-020-09909-5 · Cited by 77
Software Defect Prediction Using Stacking Generalization of Optimized Tree-Based Ensembles
Alazba, A., & Aljamaan, H. (2022) · Applied Sciences · doi:10.3390/app12094577 · Cited by 79
Transfer Convolutional Neural Network for Cross-Project Defect Prediction
Qiu, S., Xu, H., Deng, J., Jiang, S., & Lu, L. (2019) · Applied Sciences · doi:10.3390/app9132660 · Cited by 60
Predictive Models in Software Engineering: Challenges and Opportunities
Yang, Y., Xia, X., Lo, D., Bi, T., Grundy, J., & Yang, X. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3503509 · Cited by 81
“Random Forest [177] Subclass discriminant analysis (SDA) [143] Random Forest algorithm [424] naive Bayes classifiers, logistic regression classifiers, random forest classifiers [352]”
Pacheco, M., Oliva, G. A., Rajbahadur, G. K., & Hassan, A. E. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3549542 · Cited by 69
“We also note that we used Alpha centrality in lieu of the more popular Scott-Knott ESD technique(23) [42] because our data does not fully meet the assumptions of the latter ś and using unsuitable statistical tests and procedures is a common pitfall in software engineering research [36].”
MethodSelf-citationHeterogeneous Defect Prediction Through Multiple Kernel Learning and Ensemble Learning
Li, Z., Jing, X., Zhu, X., & Zhang, H. (2017) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2017.19 · Cited by 52
“A recent study [66] points out that model validation techniques may produce different bias and variances of performance estimates.”
BackgroundKernel Spectral Embedding Transfer Ensemble for Heterogeneous Defect Prediction
Tong, H., Liu, B., & Wang, S. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2939303 · Cited by 61
Palomba, F., & Tamburri, D. A. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110847 · Cited by 52
Metadata
- Authors
- Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K.
- Affiliations
- Nara Institute of Science and Technology (JP)
- Queen's University (CA)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2017-01-01
- Volume
- 43
- Issue
- 1
- Pages
- 1-18
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2474835145
- Cited by
- 574
- References
- 129
- Citations by year
- 2026: 39 · 2025: 53 · 2024: 69 · 2023: 56 · 2022: 58 · 2021: 93 · 2020: 63 · 2019: 75 · 2018: 45 · 2017: 18
- FWCI
- 106.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Funders
- JSPS
- Advancing Strategic International Networks to Accelerate the Circulation of Talented Researchers
- Interdisciplinary Global Networks for Accelerating Theory and Practice in Software Ecosystem
- JSPS Fellows (16J03360)
- Natural Sciences and Engineering Research Council of Canada (NSERC)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- fault prediction, model validation, k-fold cross-validation, software quality assurance, defect dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.