An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
The DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index.
Abstract
Defect prediction models help software quality assurance teams to allocate their limited resources to the most defect-prone modules. Model validation techniques, such as$k$-fold cross-validation, use historical data to estimate how well a model will perform in the future. However, little is known about how accurate the estimates of model validation techniques tend to be. In this paper, we investigate the bias and variance of model validation techniques in the domain of defect prediction. Analysis of 101 public defect datasets suggests that 77 percent of them are highly susceptible to producing unstable results– - selecting an appropriate model validation technique is a critical experimental design choice. Based on an analysis of 256 studies in the defect prediction literature, we select the 12 most commonly adopted model validation techniques for evaluation. Through a case study of 18 systems, we find that single-repetition holdout validation tends to produce estimates with 46-229 percent more bias and 53-863 percent more variance than the top-ranked model validation techniques. On the other hand, out-of-sample bootstrap validation yields the best balance between the bias and variance of estimates in the context of our study. Therefore, we recommend that future defect prediction studies avoid single-repetition holdout validation, and instead, use out-of-sample bootstrap validation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Biases may be balanced with variance of performance estimates using out-of-sample bootstrap techniques [139].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [139]
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
An Analysis of Transformations
Box, G. E. P., & Cox, D. R. (1964) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1964.tb00553.x · Cited by 15,052
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Research works printed but not linked: 9
- 10.29172/7c2a6982-6d72-4cd8-bba6-2fccb06a7011 · DOI · Paper page
- 10.2307/2289236 · doi:10.2307/2289236 · Paper page
- Jelihovschi, 2014, The ScottKnott Clustering Algorithm
- 10.1067/mod.2000.109032 · DOI · Paper page
- Meyer, 2014, E1071: Misc functions of the department of statistics (E1071), TU Wien
- 10.1109/ICSE.2005.1553571 · DOI · Paper page
- 2013, R Core Team, R: A language and environment for statistical computing, R foundation for statistical computing, Vienna, Austria
- 10.1007/978-3-031-01899-2 · doi:10.1007/978-3-031-01899-2 · Paper page
- Tantithamthavorn, 2015, ScottKnottESD: An R package of the Scott-Knott effect size difference (ESD) test
Cited by
Items 401–500 of 575
Local Processing of Massive Databases with R: A National Analysis of a Brazilian Social Programme
Paz, H., Maia, M., Moraes, F., Lustosa, R., da Costa, L. C. C., Macêdo, S., Barreto, M. E., & Ara, A. (2020) · Stats · doi:10.3390/stats3040028 · Cited by 4
“In this paper we consider a repeated holdout validation, which provides a better estimate once it reduces the bias, especially compared to the standard holdout validation method [40].”
MethodPredicting Bug in a Software using ANN Based Machine Learning Techniques
Rashmi, P., & Kambli, P. (2020) · International Conference for Innovation in Technology (INOCON) · doi:10.1109/inocon50539.2020.9298203 · Cited by 4
“The average of the results is considered as the final accuracy [20].”
BackgroundStudy of Learning Techniques for Effort Estimation in Object-Oriented Software Development
Shukla, S., & Kumar, S. (2022) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2022.3217570 · Cited by 6
“Recent research published by Tantithamthavorn et al. [80] reported that the K-fold CV strategy is the most unreliable and biased validation strategy.”
MethodMLinter: Learning Coding Practices from Examples—Dream or Reality?
Latappy, C., Perez, Q., Degueule, T., Falleri, J., Urtado, C., Vauttier, S., Blanc, X., & Teyton, C. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00092 · Cited by 7
“To address this threat, we use a validation inspired by the out-of-sample bootstrap validation approach [22] that is the best performing validation approach in [23].”
MethodTowards a More Reliable Interpretation of Defect Models
Jiarpakdee, J. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00084 · Cited by 7
“Prior studies investigate how defect modelling workflow impacts the performance of defect models [5–7, 18–20, 22, 23].”
BackgroundOn the need of preserving order of data when validating within-project defect classifiers
Falessi, D., Huang, J., Narayana, L., Thai, J. F., & Turhan, B. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09868-x · Cited by 75
“The rationale is that learning from the future is unrealistic [11][23].”
BackgroundMethodResultInfluentialArabic Speaker Identification System for Forensic Authentication Using K-NN Algorithm
Abdul-Wahid, S., Mahmoud, M. A., & Abdulwahid, N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-90235-3_40 · Cited by 4
Run, Forest, Run? On Randomization and Reproducibility in Predictive Software EngineeringPreprint
Liem, C. C. S., & Panichella, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08387 · Cited by 16
“This, even though a prior DP study did indicate that the random nature of sampling used for the validation can introduce critical biases [38], and different re-samplings can indeed lead to different results.”
InfluentialZhou, Y., Tang, H., Zhu, L., Ding, H., & Qian, J. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3707456 · Cited by 3
“We utilize the Scott-Knott Effect Size Difference (ESD) test to systematically classify the performance distributions of the investigated CLBI approaches into statistically distinct ranks, as discussed in the literature [44, 45].”
An Empirical Study on Release-Wise Refactoring Patterns
Noei, S., Li, H., & Zou, Y. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715734 · Cited by 13
“Ranking and clustering: To measure similarities and rank the identified release-wise refactoring patterns, we use the Scott-Knott-ESD [56, 57] test.”
Sharma, T., Misra, S., & Colomo‐Palacios, R. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11648-x · Cited by 4
“Figures 18, 19 In NASA datasets, an improvement post-FS can be seen in Accuracy (KNN, DT, TabNet), AUC-ROC (NB, DT, KNN), and AUPRC (NB, KNN, DT), whereas the gradient-boosted tree classifier shows a slight decrease in AUC-ROC and AUPRC values.”
Selection of optimal quantile protein biomarkers based on cell-level immunohistochemistry data
Yi, M., Zhan, T., Peck, A. R., Hooke, J. A., Kovatich, A. J., Shriver, C. D., Hu, H., Sun, Y., Rui, H., & Chervoneva, I. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05408-8 · Cited by 3
Bug report priority prediction using social and technical features
Huang, Z., Shao, Z., Fan, G., Yu, H., Yang, K., & Zhou, Z. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2616 · Cited by 3
Cross-Project Software Defect Prediction Based on SMOTE and Deep Canonical Correlation Analysis
Fan, X., Zhang, S., Wu, K., Wei, Z. X., & Ge, Y. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.046187 · Cited by 3
Investigating Reproducibility in Deep Learning-Based Software Fault Prediction
Mukhtar, A., Jannach, D., & Wotawa, F. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00038 · Cited by 6
“In this section, we provide the findings related to the reproducibility variables described in Table 3, considering both offline experiments and user studies.”
MethodA Validation Perspective on Random Forest for Precise Remaining Useful Life Prediction
Keren Persis, P., Sethuraman, R., Geetha, R., Tharaniya, P., Rajasekaran, R. T., & Prabhu, G. A. (2024) · International Conference on Computational Intelligence and Communication Networks (CICN) · doi:10.1109/cicn63059.2024.10847542 · Cited by 3
Predicting long time contributors with knowledge units of programming languages: an empirical study
Ahasanuzzaman, M., Oliva, G. A., & Hassan, A. E. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10655-9 · Cited by 6
“We rank the studied prediction models (KULTC, BAOLTC and KULTC+BAOLTC) using SK-ESD [12, 24, 42].”
InfluentialSelf-citationMalhotra, R., & Khan, K. (2024) · Intelligent Data Analysis · doi:10.1177/1088467x241301390 · Cited by 6
“51 The application of SMOTE for SDP has yielded a variety of opposing results.”
InfluentialA Comparison of Model Validation Techniques for Audio-Visual Speech Recognition
Seong, T. W., Ibrahim, M. Z., Arshad, N. W., & Mulvaney, D. J. (2017) · Lecture notes in electrical engineering · doi:10.1007/978-981-10-6451-7_14 · Cited by 3
“Furthermore, from previous work, the holdout method is known to be more sensitive to the quantity of data used in training, and if the number of values used was insufficient this may have also affected the accuracy available from the predictive model [17].”
MethodCombining Data Preprocessing Methods With Imputation Techniques for Software Defect Prediction
Kakkar, M., Jain, S., Bansal, A., & Grover, P. S. (2020) · IGI Global eBooks · doi:10.4018/978-1-7998-3016-0.ch081 · Cited by 5
Automatic Identification of High-Impact Bug Report by Product and Test Code Quality
Ding, J., Fan, G., Yu, H., & Huang, Z. (2022) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s021819402250036x · Cited by 3
Regularity or Anomaly? On The Use of Anomaly Detection for Fine-Grained JIT Defect Prediction
Lomio, F., Pascarella, L., Palomba, F., & Lenarduzzi, V. (2022) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa56994.2022.00049 · Cited by 5
“Moreover, the LOGO validation represents a good compromise between the bias and variance of estimates of defect prediction models (19), thus further strengthening its suitability for our case.”
MethodLambers, L., & Uchitel, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-30826-0 · Cited by 7
Predicting Defective Visual Code Changes in a Multi-Language AAA Video Game Project
Eng, K., Hindle, A., & Senchenko, A. (2023) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58846.2023.00062 · Cited by 4
“The NPSK test is a multiple comparison approach that uses hierarchical clustering to partition the set of medians of model performance into statistically distinct ranked groups that are not significantly different or have an insignificant effect size [52, 53].”
MethodJust-in-Time crash prediction for mobile apps
Wimalasooriya, C., Licorish, S. A., da Costa, D. A., & MacDonell, S. G. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10455-7 · Cited by 11
Ahmad, M. J., Goševa-Popstojanova, K., & Lutz, R. R. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10454-8 · Cited by 5
Toward the Fractal Dimension of Classes
Pan, W., Wu, W. M., Ming, H., Kim, D., Yang, Z., & Ma, Y. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688844 · Cited by 6
“We used the out-of-sample bootstrap sampling technique [115, 117] to randomly sample 𝑀 observations with replacement from the dataset of each subject system (suppose the dataset has 𝑀 observations) to create a training set, and the observations not included in the training set consist of a…”
A Step towards the Improvement in the Performance of Text Classification
Hussain, S., Mufti, M. R., Sohail, M. K., Afzal, H., Ahmad, G., & Khan, A. A. (2019) · KSII Transactions on Internet and Information Systems · doi:10.3837/tiis.2019.04.024 · Cited by 3
“We used 10-fold cross validation [31] in each experiment and compare the classifier’s performance on the constructed feature set of the proposed method and existing filter-based techniques.”
MethodA Monitoring and Threat Detection System Using Stream Processing as a Virtual Function for Big Data
Lopez, M. A., Duarte, O. C. M. B., & Pujolle, G. (2019) · Simpósio Brasileiro de Redes de Computadores e Sistemas Distribuídos (SBRC) · doi:10.5753/sbrc_estendido.2019.7789 · Cited by 5
“Holdout validation is the simplest method and shows good performance [97].”
MethodYang, X., Yu, H., Fan, G., & Yang, K. (2019) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2019-174 · Cited by 3
Toward a Smell-aware Prediction Model for CI Build Failures
Saidani, I., & Ouni, A. (2021) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew52652.2021.00017 · Cited by 5
Jadhav, R. (2020) · International Journal of Emerging Trends in Engineering Research · doi:10.30534/ijeter/2020/62862020 · Cited by 3
“There are many software technologies [8], [9] to support "log-based defect analysis" have been developed till date and for the modelling and processing of historical data, such as, " Analyze NOW" [10], "MEADEP" [11] etc. the modern capture techniques are integrated.”
MethodDeep Adversarial Learning Based Heterogeneous Defect Prediction
Sun, Y., Sun, Y., Wu, F., & Jing, X. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78609-0_28 · Cited by 3
Benchmarking cross-project defect prediction approaches with costs metricsPreprint
Herbold, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.04107 · Cited by 5
“Our first step was to determine if we can use ANOVA [16] in combination with a Scott-Knott test [60], which is a popular choice in recent defect prediction literature that compares multiple approaches to each other, e.g., [18, 23, 63, 64].”
MethodMonitoring subsurface conditions of pavement structures using Ultra-wideband radar technology
Todkar, S. S. (2019) · theses.fr (ABES) · doi:10.70675/1d45298ezbff6z478bz81c9z0d494b56a65a · Cited by 4
Imbalanced Dataset Analysis with Neural Network Model
Babu, M. C., & Pushpa, S. (2020) · Lecture Notes in Electrical Engineering · doi:10.1007/978-981-15-2612-1_9 · Cited by 2
Cross-project concurrency bug prediction using domain-adversarial neural network
Qin, F., Zheng, Z., Sui, Y., Gong, S., Shi, Z., & Trivedi, K. S. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112077 · Cited by 2
Liu, X., Zhou, Y., Lu, Z., Mei, Y., Yang, Y., Qian, J., & Zhou, Y. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2715 · Cited by 2
“1, there are differences in lexemes (specifically, lines 23, 48, and 53) compared to its counterpart in Eclipse-2.”
Don’t forget to change these functions! recommending co-changed functions in modern code review
Hong, Y., Tantithamthavorn, C. K., Thongtanunam, P., & Aleti, A. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107547 · Cited by 3
Self-citationBeyond Machine Learning: Exploring Deep Learning and Transformers for Software Defect Prediction
Dhawan, L., & Bansal, A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-3372-2_7 · Cited by 2
Optimizing Sales Forecasting in Dairy Industry Using Neural Networks
(2025) · Jordan Journal of Mechanical and Industrial Engineering · doi:10.59038/jjmie/190213 · Cited by 3
Evaluating the Interactions between class overlap and class imbalance for software defect prediction
Zhang, Y. L., Liu, N., Zhao, Y. D., Fan, J., & Gong, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129067 · Cited by 2
Dhal, S. B., Kar, D., & Capareda, S. C. (2026) · Global Energy Interconnection · doi:10.1016/j.gloei.2025.10.009 · Cited by 5
Energy-efficient large language models
Jareño, J., Aragón-Jurado, J. M., de la Torre, J. C., Ruiz, P., & Dorronsoro, B. (2026) · Future Generation Computer Systems · doi:10.1016/j.future.2026.108483 · Cited by 6
Towards Improving Adaptability of Capability Driven Development Methodology in Complex Environment
Nechkoska, R. P., Poels, G., & Manceski, G. (2018) · Lecture notes in business information processing · doi:10.1007/978-3-319-92898-2_2 · Cited by 2
Automatic Identification of High Impact Bug Report by Test Smells of Textual Similar Bug Reports
Ding, J., Fan, G., Yu, H., & Huang, Z. (2021) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs54544.2021.00056 · Cited by 2
The Impact of Parameters Optimization in Software Prediction Models
Ali, A., & Gravino, C. (2022) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa56994.2022.00041 · Cited by 3
“We apply a (nested) k-fold CV because it is a widely employed validation method in Software Engineering (used in 49% studies, followed by hold-out validation in 45%, according to Chakkrit et al. [41]).”
MethodModelling Open-Source Software Reliability Incorporating Swarm Intelligence-Based Techniques
Shatnawi, O. (2023) · International Journal of Computer Science and Information Technology · doi:10.5121/ijcsit.2023.15602 · Cited by 2
Khanna, M., & Bansal, A. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2660 · Cited by 2
“59 This was not within the scope of the study.”
Two sides of the same coin: A study on developers' perception of defects
dos Santos, G. E., Pereira, I. M., & Figueiredo, E. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2699 · Cited by 4
“Our findings suggest that Random Forest, LightGBM, Extra Trees, Gradient Boosting Machine, and KNN perform marginally better than other baseline models.”
Code Ownership: The Principles, Differences, and Their Associations with Software Quality
Thongtanunam, P., & Tantithamthavorn, C. K. (2024) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre62328.2024.00044 · Cited by 7
“Product metrics measure code characteristics in three dimensions: (1) complexity, e.g., McCabe Cyclomatic, (2) volume, e.g., lines of code, and (3) object-oriented design, e.g., coupling between object classes.”
Self-citationBIPE: A Bi-Layer Predictive Ensemble Framework for Forest Fire Susceptibility Mapping in Germany
Hu, L., Hochschild, V., Neidhardt, H., Schultz, M., Khosravani, P., & Shokati, H. (2024) · Remote Sensing · doi:10.3390/rs17010007 · Cited by 4
Enhancing Code Smell Detection Performance in Python Programming Language : A Comparative Study
Retnani, W. E. Y., Siahaan, D., Bukhori, S., Dharmawan, T., & Bayu, J. (2024) · IEEE International Conference on Electrical Engineering, Computer and Information Technology (ICEECIT) · doi:10.1109/iceecit63698.2024.10859904 · Cited by 2
“Although this method requires repetition and repeated testing, it can produce more accurate results and prevent overfitting, resulting in more reliable estimates of model performance [18].”
Zou, Y., Wang, H., Lv, H., & Zhao, S. (2025) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194025500172 · Cited by 2
Unsupervised Software Defect Prediction Through Multiview Clustering
Li, Z., Zhang, H., Jing, X., Yu, W., & Liu, Y. (2025) · IEEE Transactions on Reliability · doi:10.1109/tr.2025.3548107 · Cited by 6
“To compare their performance with supervised methods, we adopt the out-of-sample bootstrap validation [6], [59] with 100 repetitions to mitigate potential data division bias.”
Ali, M. A., Jamal, S., Wahid, N., & Ahmad, W. S. (2025) · World Development Sustainability · doi:10.1016/j.wds.2025.100218 · Cited by 5
Zhang, Z., Zhu, L., Yin, S., Hu, W., Gao, S., Chen, H., & Li, F. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00524-6 · Cited by 2
“Tantithamthavorn et al. (2016) proposed a method, called the Scott-Knott effect size difference test, can compare the performance of different feature selection and reduction methods across various classification models and datasets.”
The role of meta-learners in the adaptive selection of classifiers
Nucci, D. D., & Lucia, A. D. (2018) · IEEE Workshop on Machine Learning Techniques for Software Quality Evaluation (MaLTeSQuE) · doi:10.1109/maltesque.2018.8368452 · Cited by 2
“To this aim, we exploited the Scott-Knott ESD test [51]: this is an extension of the original Scott-Knott test [52] that (i) applies a hierarchical clustering algorithm to group together the performances of the crossproject bug prediction models experimented based on the statistically significance of the differences o…”
MethodPoster: How Do Community Smells Influence Code Smells?
Palomba, F., Tamburri, D. A., Serebrenik, A., Zaidman, A., Fontana, F. A., & Oliveto, R. (2018) · Cited by 2
Revisting the Impact of Regression Models for Predicting the Number of Defects
Wu, M., Ye, S., Li, C., Ma, Z., & Fu, Z. (2018) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2018-068 · Cited by 2
Predictive Model using SVM to Improve the Effectiveness of Direct Marketing
Sagala, N. T. M., & Permai, S. D. (2021) · International Conference on Artificial Intelligence and Data Sciences (AiDAS) · doi:10.1109/aidas53897.2021.9574440 · Cited by 2
Pattern-based validation metric for simulation models
Laili, Y., Li, Z., & Luo, Y. (2019) · Science China Information Sciences · doi:10.1007/s11432-018-9559-9 · Cited by 2
Improving Change Prediction Models with Code Smell-Related InformationPreprint
Catolino, G., Palomba, F., Fontana, F. A., Lucia, A. D., Zaidman, A., & Ferrucci, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10889 · Cited by 2
“size|measured using Cli’s Delta (or d) [39]|is negligible, so that the creation of trivial groups is avoided. To perform the test, we relied on the implementation provided by Tantithamthavorn et al. [114]. 3.4 RQ 2 - Comparison between Intensity Index and Antipattern Metrics In RQ 2 our goal was to compare the perform…”
InfluentialOn the Need of Removing Last Releases of Data When Using or Validating Defect Prediction ModelsPreprint
Ahluwalia, A., Penta, M. D., & Falessi, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.14376 · Cited by 8
“The evaluation of classifiers is performed by splitting [17] or sampling a dataset into training and testing sets in specific ways [60].”
MethodThe Threat to the Validity of Predictive Mutation Testing: The Impact of Uncovered MutantsPreprint
Aghamohammadi, A., & Mirian‐Hosseinabadi, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.11532 · Cited by 2
Does it matter who pays back Technical Debt? An empirical study of self-fixed TDPreprint
Tan, J., Feitosa, D., & Avgeriou, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05928 · Cited by 2
“As a variant of the Scott–Knott test [39], the Scott–Knott ESD test evaluates a nonnormally distributed dataset and merges any two statistically distinct groups that have a negligible effect size into a single group [38].”
MethodCode Revert Prediction with Graph Neural Networks: A Case Study at J.P. Morgan Chase
Pei, Y., Alamir, S., Dolga, R., & Shah, S. (2023) · doi:10.1145/3617572.3617879 · Cited by 2
“An empirical comparison of these methods is presented in [24].”
MethodSantana, G., Ferreira, P. H., & Fiaccone, R. L. (2022) · Brazilian Journal of Biometrics · doi:10.28951/bjb.v40i1.538 · Cited by 1
On the Impact of Draft Pull Requests on Accelerating Feedback
Harbaoui, F., Sayagh, M., & Abdalkareem, R. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00056 · Cited by 4
“We leverage permutation analysis for each model of our 100-bootstrap-based-models to obtain at the 100 ranks, which we classify using the Scott-Knott alogirhtm [15] [16] to obtain a final rank of features from the most important to the least important one.”
The Trade-off Between Data Volume and Quality in Predicting User Satisfaction in Software Projects
Radliński, Ł. (2024) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa64295.2024.00080 · Cited by 3
“0) [49], [50] to create non-overlapping partitions, i.e., ranks, of data variants.”
Simulation of Multimodal Piezoelectric Energy Harvester for Bio-Medical Applications
Raju, A., & Krishnasamy, M. (2024) · International Conference on Artificial Intelligence and Signal Processing (AISP) · doi:10.1109/aisp61711.2024.10870661 · Cited by 1
“The generator consists of a PVDF layer with a thickness of 0.1 mm and PI films with a thickness of 0.2 mm [21].”
InfluentialCross-Language Code Smell Detection via Transfer Learning
Sandouka, R., & Aljamaan, H. (2025) · Applied Sciences · doi:10.3390/app15179293 · Cited by 2
Intelligent Software Fault Prediction with Feature Optimization and Ensemble Learning
Mehta, A. (2025) · International Journal of Reliability Quality and Safety Engineering · doi:10.1142/s0218539325500391 · Cited by 1
Rechecking Recheck Requests in Continuous Integration: An Empirical Study of OpenStack
Brus, Y., Maipradit, R., Barr, E. T., & McIntosh, S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00121 · Cited by 4
“Table II shows the evaluation of traditional machine learning methods using time-based 10-fold cross-validation, avoiding single-repetition holdout due to its bias and variance issues [29].”
Self-citationSim, Y., Park, Y., & Kwon, K. (2026) · Science Advances · doi:10.1126/sciadv.aea8902 · Cited by 2
Li, G.-A., Lin, C.-Y., Dong, C.-M., Ren, L.-H., Ma, P., Wu, Y., Zhang, G., Du, X.-Y., & Zhao, Z.-R. (2026) · Petroleum Science · doi:10.1016/j.petsci.2026.01.042 · Cited by 1
Fattahi, S., Azarhoushang, B., Paknejad, M., & Kitzig-Frank, H. (2026) · Machines · doi:10.3390/machines14020196 · Cited by 3
Understanding the Amount of Changes Required for Merge Request Acceptance: An Empirical Study
Kansab, S., Sayagh, M., Bordeleau, F., & Tizghadam, A. (2025) · IEEE International Conference on Software Engineering and Service Science (ICSESS) · doi:10.1109/icsess67729.2025.11380541 · Cited by 1
“2) Correlation & Redundancy Analysis: Collinearity be-tween features can affect the interpretability of machine learning models [29, 5].”
UntrustVul: Automated Untrustworthy Alert Identification in Vulnerability Detection Models
Tung, L. N., Du, X., Neelofar, N., & Aleti, A. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3693636 · Cited by 1
“Comparison of U NTRUST V UL , baselines, and ablations using the Scott-Knott effect size difference (ESD) test [55], with colors indicating statistically distinct clusters.”
Influential闫冠龙, Li, Y., & Pan, Z. (2026) · Information · doi:10.3390/info17070658 · Cited by 2
Kondo, M., Mizuno, O., & Choi, E.-H. (2018) · International Journal of Mathematical Engineering and Management Sciences · doi:10.33889/ijmems.2018.3.2-008 · Cited by 1
“Tantithamthavorn et al. (2017) state that 100 is a sufficient value for bootstrap sampling.”
BackgroundSoftware Defect Prediction on Unlabelled Datasets: A Comparative Study
Ronchieri, E., Canaparo, M., & Belgiovine, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58802-1_25 · Cited by 1
“The Scott-Knott Effect Size Difference (ESD) test [43] is a mean comparison approach based on a hierarchical clustering analysis to partition the set of treatment means into statistically distinct groups.”
MethodA Regression Analysis Based Model for Defect Learning and Prediction in Software Development
Memon, M. A., Baloch, M.-U.-R. M., Memon, M., & Musavi, S. H. A. (2021) · Mehran University Research Journal of Engineering and Technology · doi:10.22581/muet1982.2103.15 · Cited by 1
Wavelet Support Vector Censored Regression
Maia, M., Pimentel, J. S., Ospina, R., & Ara, A. (2023) · Analytics · doi:10.3390/analytics2020023 · Cited by 1
“The measures were calculated over a repeated holdout validation with 100 repetitions and a split ratio of 70–30% of training–test, respectively [39,40].”
MethodMapping Oil Palm Plantations Using WorldView-2 Satellite Imagery and Machine Learning Algorithms
Marzuki, F., Shafri, H. Z. M., Ang, Y., Shaharum, N. S. N., Lee, Y. P., Bakar, S. A., Abidin, H., Lim, H. S., & Abdullah, R. (2023) · IOP Conference Series Earth and Environmental Science · doi:10.1088/1755-1315/1240/1/012013 · Cited by 1
“While kfold cross-validation is considered to be unbiased, it has been demonstrated that this method can generate unstable results with small samples [10].”
MethodJin, X., Li, J., Zou, Q., & Xian, Y. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0141290 · Cited by 1
“For a comprehensive explanation of the Scott-Knott ESD test, please refer to [53].”
MethodAe-RF- An Enhanced Methodology for Just In Time Software Defect Detection
Malhotra, R., & Nazari, B. A. (2020) · International Journal of Scientific Research in Computer Sciences and Engineering · doi:10.26438/ijsrcse/v8i4.127135 · Cited by 1
“Early stopping is a tool that helps us add a sizeable arbitrary number for training epochs and avoid training whenever the model's output stops improving on a hold-out [4], [34] validation dataset.”
InfluentialImproving the Condensing of Reverse Engineered Class Diagrams using Weighted Network Metrics
Pan, W., Wu, W. M., Ming, H., Kim, D., Yang, J., & Liu, R. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3643520 · Cited by 1
“To ensure the statistical robustness of the obtained results, we used the out-of-sample bootstrap sampling technique [7].”
MethodZhou, Y., Liu, X., Guo, Z., Zhou, Y., Zhang, C., & Qian, J. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2696 · Cited by 2
“We employ the ScottKnott Effect Size Difference (ESD) test to meticulously categorize distributions into statistically distinct ranks, as highlighted in references [26, 27].”
Hyperdimensional Computing Approaches in Single Cell RNA Sequencing Classification
Barmpas, P. T., Tasoulis, S. K., Georgakopoulos, S. V., & Plagianakos, V. P. (2024) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec60901.2024.10612207 · Cited by 1
“The methods were trained following their standard protocols, using ten-fold cross-validation on each dataset to ensure a fair comparison and to give each method the best opportunity to perform optimally [30].”
MethodIC‐GraF: An Improved Clustering with Graph‐Embedding‐Based Features for Software Defect Prediction
Wang, X., Lu, L., Tian, Q., & Lin, H. (2024) · IET Software · doi:10.1049/2024/8027037 · Cited by 4
A more accurate bug localization technique for bugs with multiple buggy code files
Xu, H. Y., Wang, Z., & Zou, W. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107675 · Cited by 1
MICVIF: an automatic unsupervised feature selection method for software defect prediction
Zhu, W., Li, Z., & Liu, H. (2024) · IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) · doi:10.1109/ispa63168.2024.00198 · Cited by 1
“To mitigate this threat, we utilized the non-parametric NPSKESD test [16], [27], [32] to group the various methods studied and employed the non-parametric Wilcoxon signed-rank test [19], [30] to compare the significance differences between our MICVIF approach and others at a 95% significance level.”
On the Relative Value of Feature Selection Techniques for Code Smell Detection
Zhang, Z., Zhu, L., Yin, S., Gao, S., Chen, H., Hu, W., & Li, F. (2024) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec65559.2024.00038 · Cited by 1
Singh, P., Malhotra, R., & Bansal, S. (2019) · International Conference on Cloud Computing, Data Science & Engineering (Confluence) · doi:10.1109/confluence.2019.8776946 · Cited by 1
“[14] used 12 model validation techniques using a statistical and two ML algorithms on 18 datasets.”
MethodLeveraging Machine Learning to Improve Software Reliability
Wang, S. (2019) · UWSpace (University of Waterloo) · Cited by 1
Not All Bugs Are the Same: Understanding, Characterizing, and Classifying the Root Cause of BugsPreprint
Catolino, G., Palomba, F., Zaidman, A., & Ferrucci, F. (2019) · arXiv.org · doi:10.48550/arxiv.1907.11031 · Cited by 10
“While such a strategy has recently been criticized [77], we tackled its main issue, i.e., the randomness of the splits, by running it 100 times.”
BackgroundJahanshahi, H., Çevik, M., & Başar, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03490 · Cited by 2
“Hence, we established three important inclusion criteria to filter out unreliable datasets [23]:”
Zhang, C., Cotroneo, D., Pietrantuono, R., Natella, R., Yu, X., Xie, W., Jia, K., & Xiang, J. (2023) · Research Square · doi:10.21203/rs.3.rs-2901295/v1 · Cited by 1
Leveraging Intermediate Artifacts to Improve Automated Trace Link Retrieval
Rodriguez, A. D. (2021) · DigitalCommons - CalPoly (California State Polytechnic University) · doi:10.15368/theses.2021.114 · Cited by 1
Metadata
- Authors
- Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K.
- Affiliations
- Nara Institute of Science and Technology (JP)
- Queen's University (CA)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2017-01-01
- Volume
- 43
- Issue
- 1
- Pages
- 1-18
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2474835145
- Cited by
- 574
- References
- 129
- Citations by year
- 2026: 39 · 2025: 53 · 2024: 69 · 2023: 56 · 2022: 58 · 2021: 93 · 2020: 63 · 2019: 75 · 2018: 45 · 2017: 18
- FWCI
- 106.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Funders
- JSPS
- Advancing Strategic International Networks to Accelerate the Circulation of Talented Researchers
- Interdisciplinary Global Networks for Accelerating Theory and Practice in Software Ecosystem
- JSPS Fellows (16J03360)
- Natural Sciences and Engineering Research Council of Canada (NSERC)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- fault prediction, model validation, k-fold cross-validation, software quality assurance, defect dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.