The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Reliable Accuracy Estimates from k-Fold Cross Validation
Abstract
It is popular to evaluate the performance of classification algorithms by k-fold cross validation. A reliable accuracy estimate will have a relatively small variance, and several studies therefore suggested to repeatedly perform k-fold cross validation. Most of them did not consider the correlation among the replications of k-fold cross validation, and hence the variance could be underestimated. The purpose of this study is to explore whether k-fold cross validation should be repeatedly performed for obtaining reliable accuracy estimates. The dependency relationships between the predictions of the same instance in two replications of k-fold cross validation are first analyzed for k-nearest neighbors with k = 1. Then, statistical methods are proposed to test the strength of the dependency level between the accuracy estimates resulting from two replications of k-fold cross validation. The experimental results on 20 data sets show that the accuracy estimates obtained from various replications of k-fold cross validation are generally highly correlated, and the correlation will be higher as the number of folds increases. The k-fold cross validation with a large number of folds and a small number of replications should be adopted for performance evaluation of classification algorithms.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Data Mining: Practical Machine Learning Tools and Techniques
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,437
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Mathematical Methods of Statistics.
Geary, R. C., & Cramer, H. (1947) · The Economic Journal · doi:10.2307/2226151 · Cited by 5,927
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840 · cited 2×
“That is the main reason why Dietterich [9] suggested that 2-fold cross validation should not be repeated for more than five times.”
BackgroundMethodSensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation
Rodriguez, J. D., Perez, A., & Lozano, J. A. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.187 · Cited by 1,890
Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation
Wong, T.-T. (2015) · Pattern Recognition · doi:10.1016/j.patcog.2015.03.009 · Cited by 1,562 · cited 2×
“The six books summarized in [8] do not have consistent conclusion about whether k - fold cross validation should be repeatedly performed.”
BackgroundSelf-citationHeuristics of instability and stabilization in model selection
Breiman, L. (1996) · The Annals of Statistics · doi:10.1214/aos/1032181158 · Cited by 1,146 · cited 1×
“Since the partition is random, the variance of the accuracy estimates can be large for statistical inference [1].”
BackgroundStatistical Comparisons of Classifiers over Multiple Data Sets
DemšarJanez (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248548 · Cited by 1,074
No Unbiased Estimator of the Variance of K-Fold Cross-ValidationPreprint
Bengio, Y., & Grandvalet, Y. (2003) · RePEc: Research Papers in Economics · Cited by 883
Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap
Kim, J.-H. (2009) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2009.04.009 · Cited by 792 · cited 1×
“Vanwinckelen and Blockeel [5] and Kim [6] also pointed out that repeating k -fold cross validation can stabilize the variability of accuracy estimates.”
BackgroundNo Unbiased Estimator of the Variance of K-Fold Cross-Validation
BengioYoshua, & GrandvaletYves (2004) · Journal of Machine Learning Research · doi:10.5555/1005332.1044695 · Cited by 607
Combined 5 × 2 cv F Test for Comparing Supervised Classification Learning Algorithms
Alpaydm, E. (1999) · Neural Computation · doi:10.1162/089976699300016007 · Cited by 431
Moreno-Torres, J. G., Saez, J. A., & Herrera, F. (2012) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2199516 · Cited by 306 · cited 1×
“The number of folds is generally set to be 2, 5, or 10 in exploring the characteristics of k -fold cross validation [2], [13], [14], [18].”
BackgroundDependency Analysis of Accuracy Estimates in k-Fold Cross Validation
Wong, T.-T., & Yang, N.-Y. (2017) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2740926 · Cited by 187 · cited 5×
“As pointed out by Wong and Yang [15], the (cid”
BackgroundInfluentialSelf-citationOn estimating model accuracy with repeated cross-validation
Vanwinckelen, G., & Blockeel, H. (2012) · Lirias · Cited by 156
Choosing between two learning algorithms based on calibrated tests
Bouckaert, R. (2003) · Cited by 146
Blocked 3×2 Cross-Validated t-Test for Comparing Supervised Classification Learning Algorithms
Wang, Y., Ruibo, W., Huichen, J., & Li, J. (2013) · Neural Computation · doi:10.1162/neco_a_00532 · Cited by 28
Measure for data partitioning in m × 2 cross-validation
Wang, Y., Li, J., & Li, Y. (2015) · Pattern Recognition Letters · doi:10.1016/j.patrec.2015.08.002 · Cited by 10 · cited 1×
“[12] studied the quality of data partitioning in m 2 cross validation.”
Background
Cited by
Performance of deep learning vs machine learning in plant leaf disease detection
Sujatha, R., Chatterjee, J. M., Jhanjhi, N., & Brohi, S. N. (2021) · Microprocessors and Microsystems · doi:10.1016/j.micpro.2020.103615 · Cited by 653
Hossain, E., Rana, R., Higgins, N., Soar, J., Barua, P. D., Pisani, A. R., & Turner, K. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106649 · Cited by 407
Ali, K., Shaikh, Z. A., Khan, A. A., & Laghari, A. A. (2022) · Neuroscience Informatics · doi:10.1016/j.neuri.2021.100034 · Cited by 351
Ileberi, E., Sun, Y., & Wang, Z. (2021) · IEEE Access · doi:10.1109/access.2021.3134330 · Cited by 244
PP-Net: A Deep Learning Framework for PPG-Based Blood Pressure and Heart Rate Estimation
Panwar, M., Gautam, A., Biswas, D., & Acharyya, A. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.2990864 · Cited by 237
Metadata
- Authors
- Wong, T.-T., & Yeh, P.-Y.
- Affiliations
- National Cheng Kung University (TW)
- ORCID
- Wong, Tzu-Tsung
- Venue
- IEEE Transactions on Knowledge and Data Engineering
- Published
- 2020-08-01
- Volume
- 32
- Issue
- 8
- Pages
- 1586-1594
- ISSN
- 1041-4347 (ISSN-L)
- 1558-2191
- 2326-3865
- OpenAlex
- W2942137712
- Cited by
- 1,074
- References
- 18
- Citations by year
- 2026: 178 · 2025: 292 · 2024: 248 · 2023: 178 · 2022: 102 · 2021: 56 · 2020: 18
- FWCI
- 20.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Knowledge and Data Engineering (publishedVersion)
- Funders
- Ministry of Science and Technology, Taiwan (106-2410-H-006-020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Keywords
- k-fold cross-validation, accuracy estimation, repeated cross-validation, k-nearest neighbors, classifier evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.