An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
The DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index.
Abstract
Defect prediction models help software quality assurance teams to allocate their limited resources to the most defect-prone modules. Model validation techniques, such as$k$-fold cross-validation, use historical data to estimate how well a model will perform in the future. However, little is known about how accurate the estimates of model validation techniques tend to be. In this paper, we investigate the bias and variance of model validation techniques in the domain of defect prediction. Analysis of 101 public defect datasets suggests that 77 percent of them are highly susceptible to producing unstable results– - selecting an appropriate model validation technique is a critical experimental design choice. Based on an analysis of 256 studies in the defect prediction literature, we select the 12 most commonly adopted model validation techniques for evaluation. Through a case study of 18 systems, we find that single-repetition holdout validation tends to produce estimates with 46-229 percent more bias and 53-863 percent more variance than the top-ranked model validation techniques. On the other hand, out-of-sample bootstrap validation yields the best balance between the bias and variance of estimates in the context of our study. Therefore, we recommend that future defect prediction studies avoid single-repetition holdout validation, and instead, use out-of-sample bootstrap validation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Biases may be balanced with variance of performance estimates using out-of-sample bootstrap techniques [139].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [139]
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
An Analysis of Transformations
Box, G. E. P., & Cox, D. R. (1964) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1964.tb00553.x · Cited by 15,052
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Research works printed but not linked: 9
- 10.29172/7c2a6982-6d72-4cd8-bba6-2fccb06a7011 · DOI · Paper page
- 10.2307/2289236 · doi:10.2307/2289236 · Paper page
- Jelihovschi, 2014, The ScottKnott Clustering Algorithm
- 10.1067/mod.2000.109032 · DOI · Paper page
- Meyer, 2014, E1071: Misc functions of the department of statistics (E1071), TU Wien
- 10.1109/ICSE.2005.1553571 · DOI · Paper page
- 2013, R Core Team, R: A language and environment for statistical computing, R foundation for statistical computing, Vienna, Austria
- 10.1007/978-3-031-01899-2 · doi:10.1007/978-3-031-01899-2 · Paper page
- Tantithamthavorn, 2015, ScottKnottESD: An R package of the Scott-Knott effect size difference (ESD) test
Cited by
Items 101–200 of 575
An Empirical Study of Learning to Rank Techniques for Effort-Aware Defect Prediction
Yu, X., Bennin, K. E., Liu, J., Keung, J., Yin, X., & Xu, Z. (2019) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8668033 · Cited by 55
“[77], because it has been suggested to generate the best balance between the bias and variance of training and testing datasets.”
BackgroundEffort-aware and just-in-time defect prediction with neural network
Qiao, L., & Wang, Y. (2019) · PLoS ONE · doi:10.1371/journal.pone.0211359 · Cited by 54
PerfJIT: Test-Level Just-in-Time Prediction for Performance Regression Introducing Commits
Chen, J., Shang, W., & Shihab, E. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3023955 · Cited by 63
“To find statistically distinct ranks of metrics, we perform Scott-Knott Effect Size Difference (ESD) test [82] to cluster the metrics based on both their statistical significance and effect sizes.”
MethodShahin, M., Gonsalvez, C., Whittle, J., Chen, C., Li, L., & Xia, X. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111107 · Cited by 59
Jain, S., & Saha, A. (2021) · Evolutionary Intelligence · doi:10.1007/s12065-020-00536-z · Cited by 59
Li, Z., Zhang, H., Jing, X., Xie, J., Guo, M., & Ren, J. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3204589 · Cited by 58
“Like prior defect prediction work [62], [72], [78], [81], we perform a double Scott-Knott ESD test to rank the proposed approach and the compared CPDP methods.”
MethodAn Abstract Syntax Tree Encoding Method for Cross-Project Defect Prediction
Cai, Z., Lu, L., & Qiu, S. (2019) · IEEE Access · doi:10.1109/access.2019.2953696 · Cited by 59
On the time-based conclusion stability of cross-project defect prediction models
Bangash, A. A., Sahar, H., Hindle, A., & Ali, K. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09878-9 · Cited by 51
“In a later comparison study, Tantithamthavorn et al. (2017) concluded that the choice of model validation technique for defect prediction models can also affect performance results.”
BackgroundSQAPlanner: Generating Data-Informed Software Quality Improvement Plans
Rajapaksha, D., Tantithamthavorn, C. K., Jiarpakdee, J., Bergmeir, C., Grundy, J., & Buntine, W. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3070559 · Cited by 54
Self-citationCoca: Improving and Explaining Graph Neural Network-Based Vulnerability Detection Systems
Cao, S., Sun, X., Wu, X., Lo, D., Bo, L., Li, B., & Liu, W. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639168 · Cited by 79
“For Coca 𝐸𝑥𝑝 , we set 𝛼 to 0.5 to balance factual and counterfactual reasoning.”
InfluentialA novel approach for image-based olive leaf diseases classification using a deep hybrid model
Akhal, H. E., Yahya, A. B., Moussa, N., & Alaoui, A. E. B. E. (2023) · Ecological Informatics · doi:10.1016/j.ecoinf.2023.102276 · Cited by 53
Lightweight Assessment of Test-Case Effectiveness Using Source-Code-Quality Indicators
Grano, G., Palomba, F., & Gall, H. C. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2903057 · Cited by 57
“In this work, we employed the ScottKnottESD implementation 6 provided by Tantithamthavorn et al. [94].”
MethodSimplified Deep Forest Model Based Just-in-Time Defect Prediction for Android Mobile Apps
Zhao, K., Xu, Z., Zhang, T., Tang, Y., & Yan, M. L. (2021) · IEEE Transactions on Reliability · doi:10.1109/tr.2021.3060937 · Cited by 47
“In addition, in order to conduct significance analysis among our SDF model and all the comparative methods, we employ a state-of-the-art statistic test method, called Scott–Knott Effect Size Difference (Scott–Knott ESD) [39].”
MethodMultiview Transfer Learning for Software Defect Prediction
Chen, J., Yang, Y., Hu, K., Xuan, Q., Liu, Y., & Yang, C. (2019) · IEEE Access · doi:10.1109/access.2018.2890733 · Cited by 44
“In the experiment, considering the stability of different model validation techniques for defect prediction models is different [59].”
MethodA Comprehensive Investigation of the Impact of Class Overlap on Software Defect Prediction
Gong, L., Zhang, H., Zhang, J., Wei, M., & Huang, Z. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3220740 · Cited by 49
“[67] demonstrated that the EPV values of datasets affect the performance of SDP models.”
BackgroundMethodInfluentialForrest, L. N., Ivezaj, V., & Grilo, C. M. (2021) · Psychological Medicine · doi:10.1017/s0033291721004748 · Cited by 51
“Repeated 10-fold cross-validation and traditional bootstrapping were used per recommendations (Kuhn & Johnson, 2013; Tantithamthavorn et al., 2017).”
BackgroundMethodResultInfluentialHerbold, S. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2748129 · Cited by 45
“[1], the authors propose Scott-Knott Effect Size Difference (ScottKnottESD) as an extension of the Scott-Knott test [2].”
BackgroundMethodAutomatic rain and cicada chorus filtering of bird acoustic data
Brown, A. F., Garg, S., & Montgomery, J. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.105501 · Cited by 46
On the Costs and Profit of Software Defect Prediction
Herbold, S. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2957794 · Cited by 49
“, reducing the bias through sampling approaches [10], the impact of hyper parameter tuning [11], suitable baseline comparisons [12] or general guidelines that should be considered [13].”
BackgroundImproving software bug-specific named entity recognition with deep neural network
Zhou, C., Li, B., & Sun, X. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110572 · Cited by 47
The ‘as code’ activities: development anti-patterns for infrastructure as code
Rahman, A., Farhana, E., & Williams, L. A. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09841-8 · Cited by 56
Feature Selection Model based on EEG Signals for Assessing the Cognitive Workload in Drivers
Becerra-Sánchez, P., Reyes, A., & Ibáñez, J. A. G. (2020) · Sensors · doi:10.3390/s20205881 · Cited by 41
Semantic Feature Learning via Dual Sequences for Defect Prediction
Lin, J., & Lu, L. (2021) · IEEE Access · doi:10.1109/access.2021.3051957 · Cited by 41
“A detailed introduction to the Scott-Knott ESD test is available [28].”
MethodLeoni, M. L. G., Lombardelli, L., Colombi, D., Bignami, E. G., Pergolotti, B., Repetti, F., Villani, M., Bellini, V., Rossi, T., Hálasz, G., et al. (2021) · PLoS ONE · doi:10.1371/journal.pone.0254550 · Cited by 57
Design Rule Spaces: A New Model for Representing and Analyzing Software Architecture
Cai, Y., Xiao, L., Kazman, R., Mo, R., & Feng, Q. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2797899 · Cited by 39
“[52] recently compared the bias and variances of 12 defect prediction models and found that”
Examining the Stability of Logging Statements
Kabinna, S., Shang, W., Bezemer, C., & Hassan, A. E. (2016) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2016.29 · Cited by 47
Self-citationCharacterizing and predicting blocking bugs in open source projects
Valdivia-Garcia, H., Shihab, E., & Nagappan, M. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.03.053 · Cited by 42
Minku, L. L. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09686-w · Cited by 42
Large-Scale Empirical Studies on Effort-Aware Security Vulnerability Prediction Methods
Chen, X., Zhao, Y., Cui, Z., Meng, G., Liu, Y., & Wang, Z. (2019) · IEEE Transactions on Reliability · doi:10.1109/tr.2019.2924932 · Cited by 44
Review Dynamics and Their Impact on Software Quality
Thongtanunam, P., & Hassan, A. E. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2964660 · Cited by 47
“As suggested by prior work [71], the bootstrap validation technique tends to yield less bias than the traditional k-fold cross validation techniques for logistic regression models.”
MethodSelf-citationRe-evaluating method-level bug prediction
Pascarella, L., Palomba, F., & Bacchelli, A. (2018) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2018.8330264 · Cited by 48
“has the following limitations: (1) it uses 10-fold cross-validation, which is at the risk of producing biased estimates in certain circumstances [84], (2) product metrics are considered only at the end of the time frame (while bugs are found within the time frame), (3) the number of changes and the number of bugs were…”
BackgroundOn the effectiveness of log representation for log-based anomaly detection
Wu, X., Li, H., & Khomh, F. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10364-1 · Cited by 53
“Network details can be found in our replication package 1 .”
BackgroundMethodA machine learning approach for multimodal data fusion for survival prediction in cancer patients
Νικολάου, Ν., Salazar, D., RaviPrakash, H., Gonçalves, M., Mulla, R., Burlutskiy, N., Markuzon, N., & Jacob, E. (2025) · npj Precision Oncology · doi:10.1038/s41698-025-00917-6 · Cited by 65
A new weighted naive Bayes method based on information diffusion for software defect prediction
Ji, H., Huang, S., Wu, Y., Hui, Z., & Zheng, C. (2019) · Software Quality Journal · doi:10.1007/s11219-018-9436-4 · Cited by 37
“as a learning problem in software engineering (Tantithamthavorn et al. 2017), and many”
Towards prioritizing user-related issue reports of mobile applications
Noei, E., Zhang, F., Wang, S., & Zou, Y. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09684-y · Cited by 40
TSTSS: A two-stage training subset selection framework for cross version defect prediction
Xu, Z., Li, S., Luo, X., Liu, J., Zhang, T., Tang, Y., Xu, J., Yuan, P., & Keung, J. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.03.027 · Cited by 38
Francese, R., & Yang, X. (2021) · Complex & Intelligent Systems · doi:10.1007/s40747-021-00447-1 · Cited by 49
Zhao, Z., Yang, B., Li, G., Liu, H., & Jin, Z. G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111108 · Cited by 43
“Recently, deep learning has been increasingly applied into program analysis tasks, such as program classifi-cation [1, 2, 3], software defect prediction [4, 5], and code summarization [6, 7, 8].”
MethodDealing with imbalanced data for interpretable defect prediction
Gao, Y., Zhu, Y., & Zhao, Y. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107016 · Cited by 45
The impact of feature selection techniques on effort‐aware defect prediction: An empirical study
Li, F., Lu, W., Keung, J., Yu, X., Gong, L., & Li, J. (2023) · IET Software · doi:10.1049/sfw2.12099 · Cited by 37
“Finally, we apply the Scott ‐ Knott Effect Size Difference (Scott ‐ Knott ESD) test [30] to divide the feature selection methods into different rankings.”
MethodPython code smells detection using conventional machine learning models
Sandouka, R., & Aljamaan, H. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1370 · Cited by 36
“While repeated-cross-validation repeats the cross-validations process several times and addresses the performance based on the average of all folds and repeats, it can generate more accurate results and efficiently deal with overfitting with low variance to produce more reliable model performance estimates (Tantithamt…”
BackgroundA Replication Study on Code Comprehension and Expertise using Lightweight Biometric Sensors
Fucci, D., Girardi, D., Novielli, N., Quaranta, L., & Lanubile, F. (2019) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2019.00050 · Cited by 48
“, the maximum number of different values to be evaluated for each parameter [32], [33].”
BackgroundAccelerating Continuous Integration by Caching Environments and Inferring Dependencies
Gallaba, K., Ewart, J., Junqueira, Y., & McIntosh, S. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3048335 · Cited by 54
“To check whether the acceleration levels differ in terms of build duration to a statistically significant degree, we rank the build durations of each commit in each acceleration level using the Scott-Knott ESD test [38].”
MethodSelf-citationBhutamapuram, U. S., & Sadam, R. (2021) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2021.09.010 · Cited by 47
Jiang, Z., Yang, S., Liu, Z., Xu, Y., Shen, T., Qi, S., Pang, Q., Xu, J., Liu, F., & Xu, T. (2022) · Environmental Science and Pollution Research · doi:10.1007/s11356-022-18809-8 · Cited by 35
“When the data set is limited, crossvalidation can play a greater role, usually including K-Fold cross-validation, hand-out cross validation, and leave-out cross-validation (Owen and Perry 2009; Tantithamthavorn et al. 2017).”
A Study of Redundant Metrics in Defect Prediction Datasets
Jiarpakdee, J., Tantithamthavorn, C. K., Ihara, A., & Matsumoto, K. (2016) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew.2016.30 · Cited by 37
“In order to combat any potential bias and better generalize our conclusions, we select to study 101 publicly-available defect datasets that are gathered by our recent work [8].”
BackgroundSelf-citationPredicting the number of defects in a new software version
Felix, E. A., & Lee, S. P. (2020) · PLoS ONE · doi:10.1371/journal.pone.0229131 · Cited by 36
“[46] applied the Brier score in their study to measure the gap between the estimated probability and the result achieved.”
BackgroundMethodAn Investigation of Imbalanced Ensemble Learning Methods for Cross-Project Defect Prediction
Qiu, S., Lu, L., Jiang, S., & Guo, Y. (2019) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001419590377 · Cited by 39
PatchNet: Hierarchical Deep Learning-Based Stable Patch Identification for the Linux Kernel
Hoang, T., Lawall, J., Tian, Y., Oentaryo, R. J., & Lo, D. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2952614 · Cited by 49
“We also employ Scott-Knott ESD ranking [61] to statistically compare the performance of PatchNet and the four considered approaches (i.e., PatchNet, F-NN, LS-CNN, and LPU-SVM).”
On the performance of method-level bug prediction: A negative result
Pascarella, L., Palomba, F., & Bacchelli, A. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.110493 · Cited by 53
“On the one hand, Giger et al. [46] took the change history and predicted bugs from the same time frame (which could lead to incorrect results) and used cross-validation (which have been reported as at the risk of producing biased estimates in certain circumstances [118]).”
BackgroundMethodDing, Z., Chen, J., & Shang, W. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380351 · Cited by 56
“In this study, we use the sk_esd function of the ScottKnottESD package 10 in R [47].”
MethodContext-Aware Personalized Crowdtesting Task Recommendation
Wang, J., Yang, Y., Wang, S., Chen, C., Wang, D., & Wang, Q. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3081171 · Cited by 35
The Impact of Correlated Metrics on Defect ModelsPreprint
Jiarpakdee, J., Tantithamthavorn, C. K., & Hassan, A. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10271 · Cited by 34
“ages aspects of statistical inference [22, 26, 35, 85].”
Self-citationWu, W., Wang, S., Liu, B., Shao, Y., & Xie, W. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107622 · Cited by 38
An extensive evaluation of ensemble techniques for software change prediction
Catolino, G., & Ferrucci, F. (2019) · Journal of Software Evolution and Process · doi:10.1002/smr.2156 · Cited by 36
“94 , 95 This is an effect-size aware variant of the original Scott-Knott test 96 that (1) uses hierarchical cluster analysis to partition the set of treatment means into statistically distinct groups, (2) corrects the non-normal distribution of an input data set, and (3) merges any two…”
MethodLocating Latent Design Information in Developer Discussions: A Study on Pull Requests
Viviani, G., Famelis, M., Xia, X., Janik-Jones, C., & Murphy, G. C. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2924006 · Cited by 40
“To determine the features that are the most important for the whole dataset, we take the importance values from all 10 runs and apply the Scott-Knott Effect Size Difference (ESD) test [28], [53], [62].”
MethodIoT Framework for Smart Irrigation using Machine Learning Technique
Ramya, S. R., Swetha, A., & Doraipandian, M. (2020) · Journal of Computer Science · doi:10.3844/jcssp.2020.355.363 · Cited by 33
“In order to help the farmers, overcome the uncertainty of rainfall and to improve their yield, IoT based smart irrigation system is proposed with an efficient machine learning algorithm which provides a better decision-making model Tantithamthavorn et al .”
MethodEffort-aware just-in-time defect identification in practice: a case study at Alibaba
Yan, M. L., Xia, X., Fan, Y., Lo, D., Hassan, A. E., & Zhang, X. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3417048 · Cited by 40
“Following prior studies [6, 42, 48], we apply the ScottKnott ESD (SK-ESD) test on the feature importance scores.”
MethodSelf-citationAli, A., & Gravino, C. (2021) · Science of Computer Programming · doi:10.1016/j.scico.2021.102621 · Cited by 33
Tantithamthavorn, C. K., Abebe, S. L., Hassan, A. E., Ihara, A., & Matsumoto, K. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.06.001 · Cited by 40
“We estimate the optimism of our models using the following bootstrap-derived approach [48, 55].”
MethodSelf-citationAn Adversarial Discriminative Convolutional Neural Network for Cross-Project Defect Prediction
Sheng, L., Lu, L., & Lin, J. (2020) · IEEE Access · doi:10.1109/access.2020.2981869 · Cited by 32
“Besides, we made use of the Scott–Knott ESD test [37] to analyze the experimental results of the CPDP models.”
MethodAssessing the impact of parameters tuning in ensemble based breast Cancer classification
Idri, A., Ouassif, B. E., Hosni, M., & Abnane, I. (2020) · Health and Technology · doi:10.1007/s12553-020-00453-2 · Cited by 33
A study of the performance of general compressors on log files
Yao, K., Li, H., Shang, W., & Hassan, A. E. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09822-x · Cited by 44
“Prior work (Tantithamthavorn et al. 2017) applies log-transformation on the input data of Scott-Knott to meet the assumptions.”
MethodSelf-citationSHSE: A subspace hybrid sampling ensemble method for software defect number prediction
Tong, H., Lu, W., Xing, W., 刘, 斌., & Wang, S. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106747 · Cited by 34
On effort-aware metrics for defect prediction
Çarka, J., Esposito, M., & Falessi, D. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10186-7 · Cited by 44
“Tantithamthavorn et al. (2016b) replied for a possible explanation for the results aside from researcher’ bias; however, after a few months Shepperd et al. (2018) concluded that the problem of researcher’ bias remains.”
BackgroundMethodInfluentialCharacterizing and identifying reverted commits
Yan, M. L., Xia, X., Lo, D., Hassan, A. E., & Li, S. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09688-8 · Cited by 35
Self-citationRajbahadur, G. K., Wang, S., Kamei, Y., & Hassan, A. E. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2924371 · Cited by 42
“We use the Scott-Knott ESD test to rank [23], [41], [64], [65] the features with their feature importance values.”
MethodSelf-citationParameter tuning in KNN for software defect prediction: an empirical analysis
Mabayoje, M. A., Balogun, A. O., Jibril, H. A., Atoyebi, J. O., Mojeed, H. A., & Adeyemo, V. E. (2019) · Jurnal Teknologi dan Sistem Komputer · doi:10.14710/jtsiskom.7.4.2019.121-126 · Cited by 35
Evaluating Software Defect Prediction Performance: An Updated Benchmarking StudyPreprint
Li, L., Lessmann, S., & Baesens, B. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3312070 · Cited by 37
A Deep Learning-Based Approach for the Detection of Infested Soybean Leaves
Farah, N., Drack, N., Dawel, H., & Buettner, R. (2023) · IEEE Access · doi:10.1109/access.2023.3313978 · Cited by 33
“The training set trains the model while the test set is used to estimate the performance of the model [35].”
A Comparison of Machine Learning Approaches for Detecting Misogynistic Speech in Urban Dictionary
Lynn, T., Endo, P. T., Rosati, P., da Silva, I. M. D., Santos, G. L., & Ging, D. (2019) · International Conference on Cyber Situational Awareness, Data Analytics and Assessment (CyberSA) · doi:10.1109/cybersa.2019.8899669 · Cited by 37
Multidimensional Data Modeling and Model Validation for Digital Twin Workshop
Qian, W., Guo, Y. H., Cui, K., Wu, P., Fang, W., & Liu, D. (2021) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4049634 · Cited by 32
Zhu, K., Zhang, N., Ding, W., & Jiang, C. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3359568 · Cited by 31
“In this article, we depict the boxplots with the Scott–Knott effect size difference (ESD) test [39], [40] to visual the performance differences between the RTAHC model and these baseline models.”
MethodHow do community smells influence code smells?
Palomba, F., Tamburri, D. A., Serebrenik, A., Zaidman, A., Fontana, F. A., & Oliveto, R. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3183440.3194950 · Cited by 29
“We use Gain Ratio Feature Evaluation [10] to rank the independent variables and the Scott-Knott Effect Size Difference (SK-ESD) [15] to verify the ranks.”
MethodHybrid Ensemble Learning Technique for Software Defect Prediction
Khan, M. Z. (2020) · International Journal of Modern Education and Computer Science · doi:10.5815/ijmecs.2020.01.01 · Cited by 37
“Indeed, some studies have concluded that CBMs are better than SCMs for application defect prediction [6,7].”
BackgroundContext-aware in-process crowdworker recommendation
Wang, J., Yang, Y., Wang, S., Hu, Y., Wang, D., & Wang, Q. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380380 · Cited by 29
“As reported by existing work that unbalanced data could significantly affect the model performance [45, 46], to make our dataset balanced, we randomly sample an equal number of crowdworkers (who didn’t submit bugs in the specific task) with the positive instances and label them as 0.”
MethodCylinder Pressure Prediction of An HCCI Engine Using Deep Learning
Yaşar, H., Çağıl, G., Torkul, O., & Şişci, M. (2021) · Chinese Journal of Mechanical Engineering · doi:10.1186/s10033-020-00525-4 · Cited by 30
Gallaba, K., Lamothe, M., & McIntosh, S. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510211 · Cited by 43
“Then, we apply the Scott-Knott Effect Size Difference (ESD) test [39] to cluster the action types of build steps into statistically distinct ranks based on the proportion of the build time spent on each action type.”
MethodSelf-citationService price forecasting of urban charging infrastructure by using deep stacked CNN-BiGRU network
Wang, T., Fu, L., Zhou, Y., & Gao, S. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105445 · Cited by 29
An empirical assessment of best-answer prediction models in technical Q&A sites
Calefato, F., Lanubile, F., & Novielli, N. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9642-5 · Cited by 35
“Finally, we are aware that recent work on defect prediction has shown how the choice of model validation technique (i.e., repeated cross-validation, in this case) may impact the performance estimate (Tantithamthavorn et al. 2017).”
BackgroundMethodInfluentialRobust Odor Detection in Electronic Nose Using Transfer-Learning Powered Scentformer Model
Ni, W., Wang, T., Wu, Y., Chen, L., Zeng, M., Yang, J., Hu, N., Zhang, B., Xuan, F., & Yang, Z. (2025) · ACS Sensors · doi:10.1021/acssensors.5c00630 · Cited by 29
How Practitioners Perceive Coding Proficiency
Xia, X., Wan, Z., Kochhar, P. S., & Lo, D. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00098 · Cited by 30
“As shown in Tantithamthavorn et al.’s study [34], the Scott-Knott test assumed that the data is normally distributed, which might create groups that are trivially different from one another.”
MethodPredicting Software Defects with Explainable Machine Learning
dos Santos, G. E., Figueiredo, E., Veloso, A., Viggiato, M., & Ziviani, N. (2020) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3439961.3439979 · Cited by 40
“Therefore, the ESD test unveils that out of the nine classifiers used in this ex-periment, we find six clusters for the AUC evaluation metric (the left portion of Figure 1).”
BackgroundMethodInfluentialThe Effects of Vectorization Methods on Non-Functional Requirements Classification
Amasaki, S., & Leelaprute, P. (2018) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2018.00036 · Cited by 38
“To mitigate the violation, we adopted a transform (log(F1) + 1) following [23].”
MethodInfluentialData-Driven Elicitation and Optimization of Dependencies between Requirements
Deshpande, G., Arora, C., & Ruhe, G. (2019) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2019.00055 · Cited by 31
“This validation technique has been proven as the most unbiased validation technique, which can be effectively utilized in the event of smaller training data sets to avoid overfitting [27],[20],[14].”
MethodAn empirical study of the characteristics of popular Minecraft mods
Lee, D., Rajbahadur, G. K., Lin, D., Sayagh, M., Bezemer, C., & Hassan, A. E. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09840-9 · Cited by 26
Self-citationDataset Splitting Techniques Comparison For Face Classification on CCTV Images
Nurhopipah, A., & Hasanah, U. (2020) · IJCCS (Indonesian Journal of Computing and Cybernetics Systems) · doi:10.22146/ijccs.58092 · Cited by 44
“According to [10], 36.8% average of the sample did not appear in the bootstrap sample, because the sample was taken with a replacement.”
BackgroundMethodInfluentialDEJIT: A Differential Evolution Algorithm for Effort-Aware Just-in-Time Software Defect Prediction
Yang, X., Yu, H., Fan, G., & Yang, K. (2021) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194021500108 · Cited by 29
Zhang, N., Ying, S., Zhu, K., & Zhu, D. (2021) · IET Software · doi:10.1049/sfw2.12029 · Cited by 26
“In this paper, we employ the Scott-Knott Effect Size Difference (ESD) test [61, 62] to rank twelve state-of-the-art feature extraction methods and multiple defect predictors in terms of four evaluation indicators.”
MethodFeng, S., Keung, J., Zhang, P., Xiao, Y., & Zhang, M. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106742 · Cited by 26
“We further present the Scott–Knott ESD test ranking of SMOTE-based, KFN-SMOTE-based, and M-SMOTE-based techniques on the selected classifiers across 30 datasets in terms of AUC, 𝑏𝑎𝑙𝑎𝑛𝑐𝑒 , 𝑝𝑑 , and 𝑝𝑓 .”
InfluentialCode smell detection based on supervised learning models: A survey
Zhang, Y., Ge, C., Liu, H., & Zheng, K. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127014 · Cited by 26
Tantithamthavorn, C. K. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2889160.2889256 · Cited by 28
Self-citationAutomatic detection of Feature Envy and Data Class code smells using machine learning
Škipina, M., Сливка, Ј., Luburić, N., & Kovačević, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122855 · Cited by 31
What makes a popular academic AI repository?
Fan, Y., Xia, X., Lo, D., Hassan, A. E., & Li, S. (2021) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 25
Self-citationZhang, N., Ying, S., Ding, W., Zhu, K., & Zhu, D. (2021) · Information Sciences · doi:10.1016/j.ins.2021.05.008 · Cited by 27
Zhao, K., Zhou, X., Yan, M. L., Zhang, T., Yang, D., & Li, W. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106652 · Cited by 33
“SKESD incorporates the effect size analysis, which merges the statistically different groups that have a negligible effect size into one group [61].”
MethodRevisiting the Impact of Dependency Network Metrics on Software Defect Prediction
Gong, L., Rajbahadur, G. K., Hassan, A. E., & Jiang, S. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3131950 · Cited by 35
Self-citationTraining Data Selection for Cross-Project Defection Prediction: Which Approach Is Better?
Bin, Y., Zhou, K., Lu, H., Zhou, Y., & Xu, B. (2017) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2017.49 · Cited by 27
“For each target project, we generate 60 bootstrapping samples as the test data [34].”
MethodStatement frequency coverage: A code coverage criterion for assessing test suite effectiveness
Aghamohammadi, A., Mirian‐Hosseinabadi, S., & Jalali, S. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106426 · Cited by 34
A compositional model for effort‐aware Just‐In‐Time defect prediction on android apps
Zhao, K., Zhou, X., Yan, M. L., Xue, L., Li, W., & Catolino, G. (2021) · IET Software · doi:10.1049/sfw2.12040 · Cited by 24
“To overcome these two limitations, Tantithamthavorn et al. [59] proposed an improved version, called SKESD test that applied log transforming to preprocess the results of the performance indicator and quantified the effect size by applying Cohen's delta.”
Method
Metadata
- Authors
- Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K.
- Affiliations
- Nara Institute of Science and Technology (JP)
- Queen's University (CA)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2017-01-01
- Volume
- 43
- Issue
- 1
- Pages
- 1-18
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2474835145
- Cited by
- 574
- References
- 129
- Citations by year
- 2026: 39 · 2025: 53 · 2024: 69 · 2023: 56 · 2022: 58 · 2021: 93 · 2020: 63 · 2019: 75 · 2018: 45 · 2017: 18
- FWCI
- 106.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Funders
- JSPS
- Advancing Strategic International Networks to Accelerate the Circulation of Talented Researchers
- Interdisciplinary Global Networks for Accelerating Theory and Practice in Software Ecosystem
- JSPS Fellows (16J03360)
- Natural Sciences and Engineering Research Council of Canada (NSERC)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- fault prediction, model validation, k-fold cross-validation, software quality assurance, defect dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.