An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
The DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index.
Abstract
Defect prediction models help software quality assurance teams to allocate their limited resources to the most defect-prone modules. Model validation techniques, such as$k$-fold cross-validation, use historical data to estimate how well a model will perform in the future. However, little is known about how accurate the estimates of model validation techniques tend to be. In this paper, we investigate the bias and variance of model validation techniques in the domain of defect prediction. Analysis of 101 public defect datasets suggests that 77 percent of them are highly susceptible to producing unstable results– - selecting an appropriate model validation technique is a critical experimental design choice. Based on an analysis of 256 studies in the defect prediction literature, we select the 12 most commonly adopted model validation techniques for evaluation. Through a case study of 18 systems, we find that single-repetition holdout validation tends to produce estimates with 46-229 percent more bias and 53-863 percent more variance than the top-ranked model validation techniques. On the other hand, out-of-sample bootstrap validation yields the best balance between the bias and variance of estimates in the context of our study. Therefore, we recommend that future defect prediction studies avoid single-repetition holdout validation, and instead, use out-of-sample bootstrap validation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Biases may be balanced with variance of performance estimates using out-of-sample bootstrap techniques [139].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [139]
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
An Analysis of Transformations
Box, G. E. P., & Cox, D. R. (1964) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1964.tb00553.x · Cited by 15,052
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Research works printed but not linked: 9
- 10.29172/7c2a6982-6d72-4cd8-bba6-2fccb06a7011 · DOI · Paper page
- 10.2307/2289236 · doi:10.2307/2289236 · Paper page
- Jelihovschi, 2014, The ScottKnott Clustering Algorithm
- 10.1067/mod.2000.109032 · DOI · Paper page
- Meyer, 2014, E1071: Misc functions of the department of statistics (E1071), TU Wien
- 10.1109/ICSE.2005.1553571 · DOI · Paper page
- 2013, R Core Team, R: A language and environment for statistical computing, R foundation for statistical computing, Vienna, Austria
- 10.1007/978-3-031-01899-2 · doi:10.1007/978-3-031-01899-2 · Paper page
- Tantithamthavorn, 2015, ScottKnottESD: An R package of the Scott-Knott effect size difference (ESD) test
Cited by
Items 501–575 of 575
Looking Over the Research Literature on Software Engineering from 2016 to 2018
Heradio, R., Fernández-Amorós, D., Galán, D., Cabrerizo, F. J., & Herrera‐Viedma, E. (2019) · Procedia Computer Science · doi:10.1016/j.procs.2019.12.042
Prioritizing documentation effort: Can we do better?Preprint
Liu, S., Guo, Z., Li, Y., Lu, H., Chen, L., Xu, L., Zhou, Y., & Xu, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10892
“According to [44], the out-of-sample bootstrap is the least biased model validation technique in terms of both threshold-dependent and thresholdindependent performance measures.”
MethodISC-FS: An Improved Spectral Clustering with Feature Selection for Defect Prediction.
Zhou, X., Lu, L., & Qin, Y. (2020)
A Two-stage Method of Synchronization Prediction Framework in TDD
Hsieh, C.-H., & Wang, Z. (2021) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-021-06207-0
Technical Debt Repayment in Practice
Tan, J. (2021) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.195698574 · Cited by 1
Do practitioners intentionally repay their own Technical Debt and why?
Tan, J., Feitosa, D., & Paris, A. (2021) · doi:10.26226/morressier.613b5418842293c031b5b61c
Crowd Worker Recommendation for Testing Task
Wang, Q., Chen, Z., Wang, J., & Feng, Y. (2022) · Springer eBooks · doi:10.1007/978-981-16-9643-5_6
Learning Just-in-Time Code Duplicates ExtractionPreprint
AlOmar, E. A., Ivanov, A., Kurbatova, Z., Golubev, Y., Mkaouer, M. W., Ouni, A., Bryksin, T., Nguyen, L., Kini, A., & Thakur, A. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4192441
An Evaluation of Cross-Project Defect Prediction Approaches on Cross-Personalized Defect Prediction
Amasaki, S., Aman, H., & Yokogawa, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21388-5_30
How to Make Your Deep Learning Repository Popular: A Case Study on GithubPreprint
Zhou, Y., Gong, L., Li, L., Zhang, H., Yu, X., & Keung, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4351202
Development of cognitive workload models to detect driving impairment
Sánchez, E. P. B. (2021) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · doi:10.5821/dissertation-2117-369387 · Cited by 1
An Empirical Study on Model-Agnostic Techniques for Source Code-Based Defect Prediction
Zhu, Y., Gao, Y., & Yu, Q. (2023) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194023500572 · Cited by 1
SparkSim: Performance Modeling for Resource Allocation of Spark Data Science ProjectsPreprint
Ghari, S., Fokaefs, M., Li, H., & Magnin, L. (2023) · doi:10.36227/techrxiv.24438271.v1
SparkSim: Performance Modeling for Resource Allocation of Spark Data Science ProjectsPreprint
Ghari, S., Fokaefs, M., Li, H., & Magnin, L. (2023) · doi:10.36227/techrxiv.24438271
An Empirical Study on HDP Approaches
Jing, X., Chen, H., & Xu, B. (2023) · Springer eBooks · doi:10.1007/978-981-99-2842-2_6
How do Software Engineering Researchers Use GitHub? An Empirical Study of Artifacts & ImpactPreprint
Alrashedy, K., & Binjahlan, A. (2024) · Research Square · doi:10.21203/rs.3.rs-3990204/v1
Xu, H. Y., Wang, Z., & Zou, W. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4809674
Krasniqi, R. (2024) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc2332577
Enhancing Software Defect Prediction: Exploring the Predictive Power of Two Data Flow Metrics
Roman, A., Brożek, R., & Hryszko, J. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-64182-4_13
Approach for the Optimization of Machine Learning Models for Calculating Binary Function Similarity
Horimoto, S., Lucas, K., & Bauer, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64171-8_16
“When data imbalance occurs upon splitting datasets using the holdout validation, the data imbalance can introduce potential bias in the evaluation of a model [22].”
MethodDynamic learner selection for cross-project fault prediction
Khatri, Y., & Saxena, U. R. (2024) · International Journal of Systems Assurance Engineering and Management · doi:10.1007/s13198-024-02586-3 · Cited by 1
“Further, the selection was refined to exclude the datasets with over 50% defective components, adhering to the specific robust guidelines proposed by Tantithamthavorn et al. (2016).”
Leveraging Semantic Coupling Through Class Name Embeddings for Software Change Prediction
Eken, B., & Oğur, N. B. (2024) · International Conference on Artificial Intelligence, Metaverse and Cybersecurity (ICAMAC) · doi:10.1109/icamac62387.2024.10829158
“After we collected the prediction performance values for all four classifiers, we applied the Scott-Knott ESD 3 to evaluate whether the differences between different classifiers’ performance values are statistically significant [30].”
КЛАСИФИКАЦИЯ В МАШИННОТО ОБУЧЕНИЕ. КЛАСИФИКАЦИОННИ МОДЕЛИ. МЕТОДИ И МЕТРИКИ ЗА ОЦЕНКА
Тодорова, М. (2024) · Годишник на ФТН · doi:10.46687/yt24mya
Avalos, D., & Hinojosa, S. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-78440-8_20 · Cited by 1
A Transfer Weighted Homogeneous Ensemble Method for Cross-Project Software Fault Prediction
Khatri, Y., & Saxena, U. R. (2025) · International Journal of Reliability Quality and Safety Engineering · doi:10.1142/s0218539325500093
Shatnawi, O. (2025) · WSEAS Transactions on Computers archive · doi:10.37394/23205.2025.24.7
Improving Ensemble Models for Software Defect Prediction: a study applying preprocessing techniquesPreprint
Vieira, B. P. R., & Garcia, R. E. (2025) · Research Square · doi:10.21203/rs.3.rs-7483430/v1
Alfayez, R., & Alazba, A. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70047 · Cited by 2
“The test ranks models by applying hierarchical clustering to group them into distinct clusters, where the median differences in the examined performance measure values between these clusters are statistically significant and non-negligible [46, 47].”
Influential335Machine learning models for predicting drug toxicity and side effects
Sharma, A. K., Karthik, G. L., Kamra, R., Praveena, S., & Hariram, V. (2025) · De Gruyter eBooks · doi:10.1515/9783111503202-008
On the effectiveness of network metrics on key class prediction: An empirical study
Zhou, S., Wu, W. M., Wang, J., Liu, H., & Yuan, C. (2025) · PLoS ONE · doi:10.1371/journal.pone.0334408 · Cited by 1
“To ensure the statistical robustness of the obtained results, we used the out-of-sample bootstrap sampling technique [27], which randomly samples M observations with replacement from the data set of each project (suppose the data set has M observations) to create a training set.”
AI Algorithms in Medical Diagnostics: Trends, Validation, and Challenges
Belmouss, H., & Aamouche, A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-01970-7_11
Are All Code Reviews the Same? Identifying and Assessing the Impact of Merge Request Deviations
Kansab, S., Bordeleau, F., & Tizghadam, A. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00036
“…with their description, are summarized in Table V. 3) Collinearity Analysis: To enhance model reliability and interpretability, we mitigate collinearity through correlation and redundancy analysis, following Tantithamthavorn et al. [42] and Cito et al. [9], as well as previous studies [19, 41].”
Evaluating Sound Similarity Metrics for Differentiable, Iterative Sound-Matching
Salimi, A., Hindle, A., & Zai͏̈ane, O. R. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3629237
“The second stage ranks the loss functions from best to worst using the non-parametric Scott-Knott test (NPSK) [65], [66].”
A Validation Strategy for Deep Learning Models: Evaluating and Enhancing Robustness
Nuhu, A.-R., Kebria, P. M., Hemmati, V., Lartey, B., Mahmoud, M. N., Homaifar, A., & Tunstel, E. W. (2026) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2025.3650722 · Cited by 1
“Our validation procedure builds on the conventional holdout validation (hoVa) approach [31], but it takes a distinct path and is different in several ways.”
Machine Learning-Based Software Defect Prediction Using Learning to Rank and Linear Regression
Sivasankar, R., & Dr.Sumathi, K. (2025) · International Journal of Innovative Research in Advanced Engineering · doi:10.26562/ijirae.2025.v1210.06
Ali, A., Muhammad, D., Ali, M., & Din, F. (2025) · Innovations in Systems and Software Engineering · doi:10.1007/s11334-025-00623-3
Enhancing software and learning with Serbian student feedback corpora
Vidaković, D., Luburić, N., Kovačević, A., & Сливка, Ј. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09855-y · Cited by 5
“We evaluated our solution using the out-of-sample bootstrap validation, offering a reliable measure of the model’s generalizability to unseen data (Tantithamthavorn et al., 2016 ).”
Anticipating bugs: Ticket-level bug prediction and temporal proximity effects
Prova, D. L., Gentili, E., & Falessi, D. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10771-6 · Cited by 4
Yi, M., Zhan, T., Peck, A. R., Hooke, J. A., Kovatich, A. J., Shriver, C. D., Hu, H., Sun, Y., Rui, H., & Chervoneva, I. (2023) · Figshare · doi:10.6084/m9.figshare.23733120
Yi, M., Zhan, T., Peck, A. R., Hooke, J. A., Kovatich, A. J., Shriver, C. D., Hu, H., Sun, Y., Rui, H., & Chervoneva, I. (2023) · Figshare · doi:10.6084/m9.figshare.23733123
Transformer No-Load Losses Estimation Using Machine Learning From Measured Data
Mrajca, M., Bilek, V., & Bárta, J. (2026) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2026.3660790
“For data filtering and model performance evaluation purposes, the K -fold cross-validation technique [27] was applied.”
Chen, Z., Li, Z., Zhang, H., Ren, J., & Tian, F. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00050
“Regarding the data partitioning approach, we adopted the out-of-sample-bootstrap validation [43], a resampling strategy renowned for producing performance estimates with minimal bias and robust stability.”
InfluentialPrediction of East Java province monthly sugar prices with a nonparametric regression and ARIMA
Tianda, I. M., Hasri, I. M. A., Baihaqi, M., Vebriano, T., Adri, F. M., & Chamidah, N. (2026) · AIP conference proceedings · doi:10.1063/5.0309125
Two-level automatic software optimization using cooperative co-evolutionary algorithms
Aragón-Jurado, J. M., de la Torre, J. C., Nojima, Y., Ruiz, P., & Dorronsoro, B. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.115066
Zhang, T., & Zhou, H. (2026) · International Conference on Electrical Automation and Artificial Intelligence (ICEAAI) · doi:10.1109/iceaai68945.2026.11442359
“Tantithamthavorn et al. analyzed 101 public datasets and 256 studies, finding that 77% of datasets are prone to unstable results and recommending out-of-sample bootstrap validation [16].”
Planning reliability in BIM-driven industrialized construction
León, W., Guevara, J., & Garcia-Lopez, N. P. (2026) · International Journal of Construction Management · doi:10.1080/15623599.2026.2643730
A monitoring and threat detection system using stream processing as a virtual function for Big Data
Lopez, M. A. (2018) · theses.fr (ABES) · doi:10.70675/870b46d5zf877z4400z84e7z03963aea2e73
A graph based end to end defect prediction framework.
M׳Baya, A. (2022) · theses.fr (ABES) · doi:10.70675/04cad5d8zd06bz482dzbeb8z19b24a8aa3b5 · Cited by 1
Les pratiques de code : de la documentation à la détection
Latappy, C. (2024) · theses.fr (ABES) · doi:10.70675/7ef8817cz0cdez491eza396zd5982a53b9fb
Evaluating compiler optimizations for faster and greener web browsing
Aragón-Jurado, J. M., Dorronsoro, B., & Ruiz, P. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108153
Predictive Model of E-waste Generation for Sustainable Smart Cities Using Environmental Intelligence
Delgado, J. F., Pasini, A. C., Estévez, E., & Moran, C. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-19382-7_7
Ahasanuzzaman, M., Oliva, G. A., Hassan, A. E., & Jiang, Z. M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10829-z
“To estimate the performance of our models in practice, we employ the out-of-sample bootstrap validation technique with 100 repetitions [12, 55].”
InfluentialSelf-citationNative code, greener games? Investigating energy efficiency in unity builds
Aragón-Jurado, J. M., Thawonmas, R., Ruiz, P., & Dorronsoro, B. (2026) · Entertainment Computing · doi:10.1016/j.entcom.2026.101132
Kanavalik, Y., Firsova, D., Dzeikalo, A., & Goncharov, V. (2026) · American Impact Review · doi:10.66308/air.e2026041
Quantifying SZZ-induced evaluation bias in just-in-time defect prediction
Bludau, P., & Pretschner, A. (2026) · Software Quality Journal · doi:10.1007/s11219-026-09796-x
Arabat, A., Sayagh, M., & Hassine, J. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10881-9
“…we evaluate different machine learn - ing algorithms for our two models, namely Random Forest, XGBoost, AdaBoost, Extra Trees, and Multilayer perceptron , which are widely used by prior studies (e.g., Chouchen et al. 2023 ; Zhang et al. 2015; Tantithamthavorn et al. 2017; Chen et al. 2020).”
Zhou, Y., Ding, H., Tang, H., Zhu, L., Zhou, Z., & Qian, J. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3704180
“We employ the Scott-Knott Effect Size Difference (ESD) test [20], [21] to rigorously evaluate performance differences among the compared approaches.”
A Unified Benchmark for Out-of-Distribution Detection for Autonomous Driving Systems
Li, X., Zhang, J., Keung, J., Ma, X., & Liao, Y. (2026) · International Conference/Workshop on Automation of Software Test · doi:10.1145/3793654.3793758
“We also deploy ScottKnott Effect Size Difference (ESD) test [39] to distribute a statistical analysis on the collected results, shown by using different cell colors to represent ranks in Table 5 and Table 6.”
Leveraging Commit-Size Context and Hyper Co-Change Graph Centralities for Defect Prediction
Kumar, A., Ethari, H., & Agarwal, S. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00009
“This method has been widely recommended for producing stable and unbiased estimates in defect prediction studies [28], [49], [62], [67], [68].”
Özer, G., Türkkan, M., Alkan, M., Bozoğlu, T., & Derviş, S. (2026) · Journal of Phytopathology · doi:10.1111/jph.70354
Basso, F., Fernández-Figueroa, V., Maldonado, T., Martínez, D. A., Pezoa, R., Varas, M., & Villa-Rodríguez, F. (2026) · Transportation Letters · doi:10.1080/19427867.2026.2689732
An empirical study of neural network based graph representations for software defect prediction
Li, Z., Xiang, Y., Ren, J., Zhang, H., Qi, F., & Jing, X. (2026) · Science of Computer Programming · doi:10.1016/j.scico.2026.103541
An Empirical Study on Deep Learning-based Line-Level Software Defect Prediction
Zhang, E., Jiang, Y., Zhang, T., & Tong, H. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793359
“To examine the statistical significance of performance differences among the evaluated LLDP models, we apply the Scott-Knott ESD test [29].”
Role of CI Adoption in Mobile App Success: An Empirical Study of Open-Source Android Projects
Zhou, X., Ghaleb, T. A., & Hassan, S. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793343 · Cited by 3
Software defect prediction based on fuzzy clustering weighted class association rule mining
Shao, Y., Wu, W., Wang, S., Liu, B., Yao, J., & Xie, W. (2026) · Big Data Research · doi:10.1016/j.bdr.2026.100637
Beyond Large Language Models: On the relative value of active learning for code smell detection
Chen, H., Hu, C., Liu, C., Keung, J., Ma, X., Hu, W., & Ma, C. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108302
A novel cross version software defect prediction method based on adversarial validation strategy
Wu, W., Ren, X., Liu, X., Xie, W., Ye, H., Huang, B., Yang, X., Yin, L., & Ping, G. (2026) · Array · doi:10.1016/j.array.2026.101239
Sandunika, D. M. N., Herath, A. S., & Kumara, B. T. G. S. (2026) · Moratuwa Engineering Research Conference (MERCon) · doi:10.1109/mercon71835.2026.11691285
Considering contextual semantics in domain-adversarial cross-project software defect prediction
Shen, Q., Xu, J., Wu, M., Lou, J., Fei, J., & Zhu, S. (2026) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-026-03220-3
From Defect Prediction to Defect Triage: Evaluating Accepted Risk and Review Effort With CReST-DP
Alsawalqah, H., Abadleh, A., Al-Shboul, B. A., Al-Tawil, M., Aljarah, I., Alshamaileh, Y., & Alaydaa, M. S. (2026) · IEEE Access · doi:10.1109/access.2026.3740992
“Later work compared learners, feature sets, and evaluation procedures at larger scale, showing that classifier choice, dataset properties, validation protocol, and tuning decisions can substantially affect reported performance [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27].”
Mendoza-Valdés, J., & López, C. Q. (2026) · Journal on Computing and Cultural Heritage · doi:10.1145/3850157
“We execute pairwise Wilcoxon signed-rank tests [52] with Bonferroni correction on per-image metrics, supplemented by Cliff’s 𝛿 non-parametric effect sizes and Scott–Knott Effect Size Difference (ESD) hierarchical grouping [46].”
Rangari, A. J., Mishra, L. N., & Senapati, B. (2026) · Technologies · doi:10.3390/technologies14100657
Radliński, Ł. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02898-9
Metadata
- Authors
- Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K.
- Affiliations
- Nara Institute of Science and Technology (JP)
- Queen's University (CA)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2017-01-01
- Volume
- 43
- Issue
- 1
- Pages
- 1-18
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2474835145
- Cited by
- 574
- References
- 129
- Citations by year
- 2026: 39 · 2025: 53 · 2024: 69 · 2023: 56 · 2022: 58 · 2021: 93 · 2020: 63 · 2019: 75 · 2018: 45 · 2017: 18
- FWCI
- 106.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Funders
- JSPS
- Advancing Strategic International Networks to Accelerate the Circulation of Talented Researchers
- Interdisciplinary Global Networks for Accelerating Theory and Practice in Software Ecosystem
- JSPS Fellows (16J03360)
- Natural Sciences and Engineering Research Council of Canada (NSERC)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- fault prediction, model validation, k-fold cross-validation, software quality assurance, defect dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.