The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Reliable Accuracy Estimates from k-Fold Cross Validation
Abstract
It is popular to evaluate the performance of classification algorithms by k-fold cross validation. A reliable accuracy estimate will have a relatively small variance, and several studies therefore suggested to repeatedly perform k-fold cross validation. Most of them did not consider the correlation among the replications of k-fold cross validation, and hence the variance could be underestimated. The purpose of this study is to explore whether k-fold cross validation should be repeatedly performed for obtaining reliable accuracy estimates. The dependency relationships between the predictions of the same instance in two replications of k-fold cross validation are first analyzed for k-nearest neighbors with k = 1. Then, statistical methods are proposed to test the strength of the dependency level between the accuracy estimates resulting from two replications of k-fold cross validation. The experimental results on 20 data sets show that the accuracy estimates obtained from various replications of k-fold cross validation are generally highly correlated, and the correlation will be higher as the number of folds increases. The k-fold cross validation with a large number of folds and a small number of replications should be adopted for performance evaluation of classification algorithms.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Data Mining: Practical Machine Learning Tools and Techniques
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,437
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Mathematical Methods of Statistics.
Geary, R. C., & Cramer, H. (1947) · The Economic Journal · doi:10.2307/2226151 · Cited by 5,927
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840 · cited 2×
“That is the main reason why Dietterich [9] suggested that 2-fold cross validation should not be repeated for more than five times.”
BackgroundMethodSensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation
Rodriguez, J. D., Perez, A., & Lozano, J. A. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.187 · Cited by 1,890
Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation
Wong, T.-T. (2015) · Pattern Recognition · doi:10.1016/j.patcog.2015.03.009 · Cited by 1,562 · cited 2×
“The six books summarized in [8] do not have consistent conclusion about whether k - fold cross validation should be repeatedly performed.”
BackgroundSelf-citationHeuristics of instability and stabilization in model selection
Breiman, L. (1996) · The Annals of Statistics · doi:10.1214/aos/1032181158 · Cited by 1,146 · cited 1×
“Since the partition is random, the variance of the accuracy estimates can be large for statistical inference [1].”
BackgroundStatistical Comparisons of Classifiers over Multiple Data Sets
DemšarJanez (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248548 · Cited by 1,074
No Unbiased Estimator of the Variance of K-Fold Cross-ValidationPreprint
Bengio, Y., & Grandvalet, Y. (2003) · RePEc: Research Papers in Economics · Cited by 883
Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap
Kim, J.-H. (2009) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2009.04.009 · Cited by 792 · cited 1×
“Vanwinckelen and Blockeel [5] and Kim [6] also pointed out that repeating k -fold cross validation can stabilize the variability of accuracy estimates.”
Background
Cited by
Performance of deep learning vs machine learning in plant leaf disease detection
Sujatha, R., Chatterjee, J. M., Jhanjhi, N., & Brohi, S. N. (2021) · Microprocessors and Microsystems · doi:10.1016/j.micpro.2020.103615 · Cited by 653
Hossain, E., Rana, R., Higgins, N., Soar, J., Barua, P. D., Pisani, A. R., & Turner, K. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106649 · Cited by 407
Ali, K., Shaikh, Z. A., Khan, A. A., & Laghari, A. A. (2022) · Neuroscience Informatics · doi:10.1016/j.neuri.2021.100034 · Cited by 351
Ileberi, E., Sun, Y., & Wang, Z. (2021) · IEEE Access · doi:10.1109/access.2021.3134330 · Cited by 244
PP-Net: A Deep Learning Framework for PPG-Based Blood Pressure and Heart Rate Estimation
Panwar, M., Gautam, A., Biswas, D., & Acharyya, A. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.2990864 · Cited by 237
Natras, R., Soja, B., & Schmidt, M. (2022) · Remote Sensing · doi:10.3390/rs14153547 · Cited by 173
“For reliable performance evaluation, a large number of folds should be adopted [44].”
BackgroundImproved Feature Selection Model for Big Data Analytics
El-Hasnony, I. M., Barakat, S. I., Elhoseny, M., & Mostafa, R. R. (2020) · IEEE Access · doi:10.1109/access.2020.2986232 · Cited by 171
“On average, the results of different data rounds are statistically reliable results [42].”
ResultMachine Learning for Perovskite Solar Cells and Component Materials: Key Technologies and Prospects
Liu, Y., Tan, X., Liang, J., Han, H., Xiang, P., & Yan, W. (2023) · Advanced Functional Materials · doi:10.1002/adfm.202214271 · Cited by 174
Predicting students’ performance in e-learning using learning process and behaviour data
Qiu, F., Zhang, G., Sheng, X., Jiang, L., Zhu, L., Xiang, Q., Jiang, B., & Chen, P.-K. (2022) · Scientific Reports · doi:10.1038/s41598-021-03867-8 · Cited by 161
Optimized hybrid ensemble learning approaches applied to very short-term load forecasting
Yamasaki, M., Freire, R. Z., Seman, L. O., Stefenon, S. F., Mariani, V. C., & Coelho, L. D. S. (2023) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2023.109579 · Cited by 171
Internet of things-enabled real-time health monitoring system using deep learning
Wu, X., Liu, C., Wang, L., & Bilal, M. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06440-6 · Cited by 148
Hoang, N., & Van-Duc, T. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.104767 · Cited by 165
Machine Learning Approach to Support the Detection of Parkinson’s Disease in IMU-Based Gait Analysis
Trabassi, D., Serrao, M., Varrecchia, T., Ranavolo, A., Coppola, G., Icco, R. D., Tassorelli, C., & Castiglia, S. F. (2022) · Sensors · doi:10.3390/s22103700 · Cited by 165
“To obtain the k models and performance estimates, this technique is repeated k times [73].”
MethodAlsbrooks, K., & Hoerauf, K. H. (2022) · PLoS ONE · doi:10.1371/journal.pone.0276814 · Cited by 146
EEG Signal Processing and Supervised Machine Learning to Early Diagnose Alzheimer’s Disease
Pirrone, D., Weitschek, E., Paolo, P. D., Salvo, S. D., & Cola, M. C. D. (2022) · Applied Sciences · doi:10.3390/app12115413 · Cited by 124
“Consequently, the data training is segmented into k subgroups [40].”
MethodOkoye, K., Arrona-Palacios, A., Camacho-Zuñiga, C., Achem, J. A. G., Escamilla, J., & Hosseini, S. (2022) · Education and Information Technologies · doi:10.1007/s10639-021-10751-5 · Cited by 112
Emotion Recognition from ECG Signals Using Wavelet Scattering and Machine Learning
Sepúlveda, A., Castillo, F., Palma, C., & Rodriguez-Fernández, M. (2021) · Applied Sciences · doi:10.3390/app11114945 · Cited by 115
“Given the large amount of data, k-fold cross-validation [49] with k = 10 was used.”
MethodPredicting maritime accident risk using Automated Machine Learning
Munim, Z. H., Sørli, M. A., Kim, H., & Alon, I. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110148 · Cited by 119
Elazab, A., Wang, C., Abdel-Aziz, M. H., Zhang, J., Gu, J. J., Górriz, J. M., Zhang, Y., & Chang, C. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124780 · Cited by 126
Nillmani, Sharma, N., Saba, L., Khanna, N. N., Kalra, M. K., Fouda, M. M., & Suri, J. S. (2022) · Diagnostics · doi:10.3390/diagnostics12092132 · Cited by 101
“The 5‐fold cross‐validation is the most popular protocol, where 80% of images are used for training and 20% for testing the model [51,52].”
BackgroundMethodNaser, M. A., Majeed, A. A., Alsabah, M. Q., Al-Shaikhli, T. R., & Kaky, K. M. (2024) · Algorithms · doi:10.3390/a17020078 · Cited by 164
“Cross-validation techniques, such as k -fold cross-validation, split the dataset into multiple subsets, training the model on different portions and testing it on others [185,186].”
Gan, C., Feng, Q., & Zhang, Z. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.01.024 · Cited by 106
Machine learning applications in nanomaterials: Recent advances and future perspectives
Yang, L., Wang, H., Leng, D., Fang, S., Yang, Y., & Du, Y. (2024) · Chemical Engineering Journal · doi:10.1016/j.cej.2024.156687 · Cited by 111
Prediction of malaria incidence using climate variability and machine learning
Odu, N., Prasad, R., & Clement, O. (2021) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2020.100508 · Cited by 95
Machine learning-enabled optimization of extrusion-based 3D printing
Rahmani Dabbagh, S., Ozcan, O., & Tasoglu, S. (2022) · Methods · doi:10.1016/j.ymeth.2022.08.002 · Cited by 94
A feature selection approach based on NSGA-II with ReliefF
Xue, Y., Zhu, H., & Neri, F. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.109987 · Cited by 100
Kumar, C., Mubvumba, P., Huang, Y., Dhillon, J. S., & Reddy, K. N. (2023) · Agronomy · doi:10.3390/agronomy13051277 · Cited by 107
“Considering the small number of training samples (i.e., sixteen for each treatment) in this study, which limits the training and testing data split, a K-fold cross-validation method is an e ff ective approach for the performance evaluation of the di ff erent ML models [71].”
Ranjbarzadeh, R., Zarbakhsh, P., Caputo, A., Tırkolaee, E. B., & Bendechache, M. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107723 · Cited by 115
Review of machine learning approaches for predicting mechanical behavior of composite materials
Sharma, H., Arora, G., Singh, M. K., Ayyappan, V., Bhowmik, P., Rangappa, S. M., & Siengchin, S. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07616-8 · Cited by 118
“Even if processing capacity allows for the use of additional folds, Wong et al. [123] showed that ten-fold cross-validation is the most effective approach for model validation for real-world datasets.”
Evaluating the Effectiveness of Text Pre-Processing in Sentiment Analysis
Palomino, M. A., & Aider, F. (2022) · Applied Sciences · doi:10.3390/app12178765 · Cited by 190
“Thus, we have applied cross-validation [79].”
Random kernel k-nearest neighbors regression
Srisuradetchai, P., & Suksrikran, K. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1402384 · Cited by 114
Prediction of the severity of marine accidents using improved machine learning
Feng, Y., Wang, X., Chen, Q., Yang, Z., Wang, J., Li, H., Xia, G., & Liu, Z. (2024) · Transportation Research Part E Logistics and Transportation Review · doi:10.1016/j.tre.2024.103647 · Cited by 91
Jin, X., Gong, W.-T., Kong, J., Bai, Y., & Su, T. (2022) · Entropy · doi:10.3390/e24030335 · Cited by 83
“The commonly used cross-validation methods in machine learning are Monte Carlo simulation [50] and K-fold cross-validation [51].”
MethodAminpour, M., Alaie, R., Khosravi, S., Kardani, N., Moridpour, S., & Nazem, M. (2022) · Computers and Geotechnics · doi:10.1016/j.compgeo.2022.105094 · Cited by 102
Zhang, Z., Min, H., Guo, H., Yu, Y., Sun, W., Jiang, J., & Zhao, H. (2023) · Journal of Energy Storage · doi:10.1016/j.est.2023.107063 · Cited by 88
Rajab, A. D., Farman, H., Islam, N., Syed, D., Elmagzoub, M. A., Shaikh, A., Akram, M. I., & Alrizq, M. (2023) · Water · doi:10.3390/w15223970 · Cited by 81
Bhandari, M., Yogarajah, P., Kavitha, M. S., & Condell, J. V. (2023) · Applied Sciences · doi:10.3390/app13053125 · Cited by 100
“The final averaged performance of the CT dataset was reported using K = 10 [59] for the separate, random training/testing assignments.”
Morais, L. B. S., Áquila, G., Faria, V., Lima, L. M. M., Lima, J. W. M., & de Queiroz, A. R. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.121439 · Cited by 83
Machine Learning in Perovskite Solar Cells: Recent Developments and Future Perspectives
Bansal, N., Mishra, S., Dixit, H., Porwal, S., Singh, P., & Singh, T. (2023) · Energy Technology · doi:10.1002/ente.202300735 · Cited by 86
Preetha, R., Priyadarsini, M. J. P., & Nisha, J. S. (2024) · IEEE Access · doi:10.1109/access.2024.3442979 · Cited by 93
“To ensure that our model does not suffer from overfitting and to validate its performance, we performed K-Fold Cross-Validation [58] using 5-fold and 10-fold splits.”
MethodApplications of UAS in Crop Biomass Monitoring: A Review
Wang, T., Liu, Y., Wang, M., Fan, Q., Tian, H., Qiao, X., & Li, Y. (2021) · Frontiers in Plant Science · doi:10.3389/fpls.2021.616689 · Cited by 100
“The value of folds can be large and the value of replications should be small if k-fold cross validation is applied in the classification algorithms (Wong and Yeh, 2020).”
MethodAn adaptive deep-learning load forecasting framework by integrating transformer and domain knowledge
Gao, J., Chen, Y., Hu, W., & Zhang, D. (2023) · Advances in Applied Energy · doi:10.1016/j.adapen.2023.100142 · Cited by 94
A Transfer Residual Neural Network Based on ResNet-50 for Detection of Steel Surface Defects
Zhang, L., Bian, Y., Jiang, P., & Zhang, F. (2023) · Applied Sciences · doi:10.3390/app13095260 · Cited by 91
Efficient framework for brain tumor detection using different deep learning techniques
Taher, F., Shoaib, M. R., Emara, H. M., Abdelwahab, K. M., El‐Samie, F. E. A., & Haweel, M. T. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.959667 · Cited by 78
“As a result, stratified k -folds and a variety of performance measures may be employed to help in the development of a reliable deep-learning-based model (41).”
BackgroundMS-FRCNN: A Multi-Scale Faster RCNN Model for Small Target Forest Fire Detection
Zhang, L., Wang, M., Ding, Y., & Bu, X. (2023) · Forests · doi:10.3390/f14030616 · Cited by 70
“In order to verify the effectiveness of the various improvement strategies adopted by the MS-FRCNN model compared to the Faster RCNN model, ablation experiments were conducted in this section.”
BackgroundMethodInfluentialSingh, S., Patro, S. K., & Parhi, S. K. (2023) · Asian Journal of Civil Engineering · doi:10.1007/s42107-023-00698-y · Cited by 69
Kokkotis, C., Ntakolia, C., Moustakidis, S., Giakas, G., & Tsaopoulos, D. E. (2022) · Physical and Engineering Sciences in Medicine · doi:10.1007/s13246-022-01106-6 · Cited by 68
“For the experimental evaluation, a repeated strati ed 5-fold cross validation was used [38].”
MethodClassification of Contaminated Insulators Using k-Nearest Neighbors Based on Computer Vision
Corso, M. P., Perez, F. L., Stefenon, S. F., Yow, K., Ovejero, R. G., & Leithardt, V. R. Q. (2021) · Computers · doi:10.3390/computers10090112 · Cited by 64
Enhancing the Evaluation and Interpretability of Data-Driven Air Quality Models
Gu, J., Yang, B., Brauer, M. D., & Zhang, K. M. (2020) · Atmospheric Environment · doi:10.1016/j.atmosenv.2020.118125 · Cited by 65
Yan, J., He, Z., & He, S. (2023) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2023.109141 · Cited by 69
Shen, Y., Wu, L., & Liang, S. (2022) · Engineering Failure Analysis · doi:10.1016/j.engfailanal.2022.106647 · Cited by 61
COVID Mortality Prediction with Machine Learning Methods: A Systematic Review and Critical Appraisal
Bottino, F., Tagliente, E., Pasquini, L., Napoli, A. D., Lucignani, M., Figà-Talamanca, L., & Napolitano, A. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11090893 · Cited by 69
“suggested to repeat K-fold cross-validation several times in order to obtain reliable accuracy [66].”
BackgroundA multimodal fusion network with attention mechanisms for visual–textual sentiment analysis
Gan, C., Fu, X., Feng, Q., Zhu, Q., Cao, Y., & Zhu, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122731 · Cited by 84
Ashik, F. R., Sreezon, A. I. Z., Rahman, M. H., Zafri, N. M., & Labib, S. M. (2024) · Journal of Transport Geography · doi:10.1016/j.jtrangeo.2024.103828 · Cited by 60
Deep learning pose estimation for multi-cattle lameness detection
Barney, S., Dlay, S. S., Crowe, A. M., Kyriazakis, I., & Leach, M. C. (2023) · Scientific Reports · doi:10.1038/s41598-023-31297-1 · Cited by 55
Eghtesad, A., Luo, Q., Shang, S., Lebensohn, R. A., Knežević, M., Liu, Z., & Beese, A. M. (2023) · International Journal of Plasticity · doi:10.1016/j.ijplas.2023.103646 · Cited by 63
Massaoudi, M., Refaat, S. S., Abu‐Rub, H., Chihi, I., & Oueslati, F. S. (2020) · Energies · doi:10.3390/en13205464 · Cited by 57
“Moreover, K-fold cross validation technique is used in this paper as a reliable means to evaluate the proposed model accuracy to an unknown testing data [43].”
Feature grouping and selection: A graph-based approach
Zheng, L., Chao, F., Parthaláin, N. M., Zhang, D., & Shen, Q. (2020) · Information Sciences · doi:10.1016/j.ins.2020.09.022 · Cited by 53
Zhou, Z., Wen, Z., Cui, J., Luan, H., Chen, Y., & Lin, D. (2025) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2025.110885 · Cited by 55
Probabilistic Time-Varying Parameter Identification for Load Modeling: A Deep Generative Approach
Khodayar, M., & Wang, J. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.2971014 · Cited by 53
“The optimal values of hyperparameters, including the LSTM’s load data time lag k ξ = 3, the measurement time lag k M = 4, and the number of hidden layers for the encoder ANN L Q = 3 and decoder ANN L P = 2, are determined by the tenfold cross-validation method [26].”
MethodMai, J., Tian, L., Xu, P., Lian, Z., Li, M., & Lu, W.-C. (2022) · Dyes and Pigments · doi:10.1016/j.dyepig.2022.110647 · Cited by 55
Yan, J., He, Z., & He, S. (2022) · Computers & Industrial Engineering · doi:10.1016/j.cie.2022.108559 · Cited by 55
Does cybersecurity risk stifle corporate innovation activities?
Wang, J., Ho, C. Y., & Shan, Y. G. (2023) · International Review of Financial Analysis · doi:10.1016/j.irfa.2023.103028 · Cited by 55
Sadaghat, B., Afzal, S., & Khiavi, A. J. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123934 · Cited by 55
Predicting Dementia With Prefrontal Electroencephalography and Event-Related Potential
Doan, D. N. T., Ku, B. U., Choi, J.-M., Oh, M., Kim, K., Cha, W., & Kim, J. U. (2021) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2021.659817 · Cited by 51
“In the medical sciences, a cross-validation approach is being increasingly adopted to obtain an unbiased prediction accuracy with high reliability (Wong and Yeh, 2020).”
MethodLiang, D., & Xue, F. (2023) · Computers in Industry · doi:10.1016/j.compind.2023.103883 · Cited by 55
“Such a method is superior to split sample validation (Ellis & Mookim 2013; Wong & Yeh 2020; Yuan et al. 2020) because using only a single split of full samples is sensitive to the particular split sample created.”
Forecasting of compound ocean-fluvial floods using machine learning
Moradian, S., AghaKouchak, A., Gharbia, S. S., Broderick, C., & Olbert, A. I. (2024) · Journal of Environmental Management · doi:10.1016/j.jenvman.2024.121295 · Cited by 54
Predicting energy performances of buildings' envelope wall materials via the random forest algorithm
Hussien, A. A., Khan, W., Hussain, A. J., Liatsis, P., Al-Shamma'a, A. I., & Al‐Jumeily, D. (2023) · Journal of Building Engineering · doi:10.1016/j.jobe.2023.106263 · Cited by 54
Wu, Y., Pieralisi, R., Sandoval, G. F. B., López-Carreño, R.-D., & Pujadas, P. (2024) · Construction and Building Materials · doi:10.1016/j.conbuildmat.2024.137619 · Cited by 47
Gao, Q., Shi, Y., Pettit, C. J., & Han, H. (2022) · Land Use Policy · doi:10.1016/j.landusepol.2022.106409 · Cited by 54
Agand, P., Kennedy, A., Harris, T. A. L. V., Bae, C., Chen, M., & Park, E. J. (2023) · Ocean Engineering · doi:10.1016/j.oceaneng.2023.115271 · Cited by 58
“Specifically, one-tenth of the data was randomly selected for each iteration of cross-validation Wong and Yeh (2019).”
MethodKumar, A., Dhanka, S., Sharma, A., Bansal, R., Fahlevi, M., Rabby, F., & Aljuaid, M. M. (2025) · Scientific Reports · doi:10.1038/s41598-025-09957-1 · Cited by 85
“Tenfold cross-validation is particularly advantageous in providing a comprehensive assessment due to its balance between computational efficiency and reliable estimation of model performance 27 .”
Toward Modeling Psychomotor Performance in Karate Combats Using Computer Vision Pose Estimation
Echeverría, J., & Santos, O. C. (2021) · Sensors · doi:10.3390/s21248378 · Cited by 57
Fault Diagnosis of Rolling Bearing Under Variable Working Conditions Based on CWT and T-ResNet
Diao, N., Wang, Z., Ma, H., & Yang, W. (2022) · Journal of Vibration Engineering & Technologies · doi:10.1007/s42417-022-00780-w · Cited by 45
“Finally, faults with different extents and distributions of the same part are regarded as the same type, and the model is evaluated with a fivefold cross-validation [14] method.”
MethodManan, A., Zhang, P., Ahmad, S., & Ahmad, J. E. (2024) · Anti-Corrosion Methods and Materials · doi:10.1108/acmm-12-2023-2935 · Cited by 44
Protein structure prediction via deep learning: an in-depth review
Meng, Y., Zhang, Z., Zhou, C., Tang, X., Hu, X., Tian, G., Yang, J., & Yao, Y. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1498662 · Cited by 54
“The most commonly used cross-validation methods include hold-out cross-validation (Ziggah et al., 2019), k-fold cross-validation (Wong and Yeh, 2019), and leave-one-out cross-validation (LOOCV) Magnusson et al. (2020).”
Khan, S. A., Khan, M., Iqbal, N., Khan, S. A., & Chou, K. (2020) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2020.104056 · Cited by 46
Addressing the Cold-Start Problem in Recommender Systems Based on Frequent Patterns
Panteli, A., & Boutsinas, B. (2023) · Algorithms · doi:10.3390/a16040182 · Cited by 43
“Then, every fold in turn plays the role of testing the model induced from the other k-1 folds [49].”
BackgroundDas, R., Ahmed, W., Sharma, K., Hardey, M., Dwivedi, Y. K., Zhang, Z., Apostolidis, C., & Filieri, R. (2024) · European Journal of Operational Research · doi:10.1016/j.ejor.2024.03.008 · Cited by 58
Li, J., Qian, J., Chen, J., Ruiz-García, L., Dong, C., Chen, Q., Liu, Z., Xiao, P., & Zhao, Z. (2024) · Comprehensive Reviews in Food Science and Food Safety · doi:10.1111/1541-4337.70082 · Cited by 56
An Empirical Approach for Extreme Behavior Identification through Tweets Using Machine Learning
Sharif, W., Mumtaz, S., Shafiq, Z., Riaz, O., Ali, T., Husnain, M., & Choi, G. S. (2019) · Applied Sciences · doi:10.3390/app9183723 · Cited by 49
“We, therefore adopted, to use a K-fold cross-validation method [52,53] where we split data into K parts.”
Yik, B. J., Dood, A. J., de Arellano, D. C., Fields, K. B., & Raker, J. R. (2021) · Chemistry Education Research and Practice · doi:10.1039/d1rp00111f · Cited by 53
El‐Hasnony, I. M., Elhoseny, M., & Tarek, Z. (2021) · Expert Systems · doi:10.1111/exsy.12786 · Cited by 41
Hoang, N., Tran, V.-D., & Huynh, T. (2025) · Sensors · doi:10.3390/s25041169 · Cited by 51
“All hyper-parameters of the machine learning models are specified by the cross-validation processes [62].”
Forssten, M. P., Bass, G. A., Ismail, A. M., Mohseni, S., & Cao, Y. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11080727 · Cited by 45
“During the model training stage, K-fold cross-validation was used [29].”
MethodChen, Y., Wang, Y., Li, L., Cui, Y., Duan, X., & Long, D. (2024) · Water Resources Research · doi:10.1029/2023wr036450 · Cited by 46
“Ten‐ fold cross‐validation stands as a dependable method for assessing machine learning model accuracy (Wong & Yeh, 2020).”
ProtInteract: A deep learning framework for predicting protein–protein interactions
Soleymani, F., Paquet, E., Viktor, H. L., Michalowski, W., & Spinello, D. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.01.028 · Cited by 44
Optimizing HCV Disease Prediction in Egypt: The hyOPTGB Framework
Elshewey, A. M., Shams, M. Y., Tawfeek, S. M., Alharbi, A. H., Ibrahim, A. F., Abdelhamid, A. A., Eid, M. M., Khodadadi, N., Abualigah, L., Khafaga, D. S., et al. (2023) · Diagnostics · doi:10.3390/diagnostics13223439 · Cited by 38
“This process is repeated k times, with each fold used once for validation and the remaining folds used The steps for Min-Max normalization are as follows: for training the model [44].”
MethodWang, L., Liu, H., Zhang, F., Guo, L., & Chen, Z. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105489 · Cited by 37
Hoang, N., Pham, P.-A.-H., Huynh, T., Cao, M.-T., & Bui, D. T. (2024) · Earth Science Informatics · doi:10.1007/s12145-024-01582-2 · Cited by 41
“Additionally, the suitable hyper-parameters of the three machine learning models were identified by five-fold cross-validation processes (Wong and Yeh 2020).”
Individualised Responsible Artificial Intelligence for Home-Based Rehabilitation
Vourganas, I., Stanković, V., & Stanković, L. (2020) · Sensors · doi:10.3390/s21010002 · Cited by 45
“The accuracy of all the models was evaluated using an 80% training and 20% testing dataset split and fivefold cross validation [91].”
MethodA wrapper methodology to learn interval-valued fuzzy rule-based classification systems
Sanz, J., & Bustince, H. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107249 · Cited by 41
Machine Learning‐Enhanced Triboelectric Sensing Application
Shang, R., Chen, H., Cai, X., Shi, X., Yang, Y., Wei, X., Wang, J., & Xu, Y. F. (2024) · Advanced Materials Technologies · doi:10.1002/admt.202301316 · Cited by 39
Si, B., Ni, Z., Xu, J., Li, Y., & Liu, F. (2024) · Case Studies in Thermal Engineering · doi:10.1016/j.csite.2024.104124 · Cited by 36
Distribution Prediction of Strategic Flight Delays via Machine Learning Methods
Wang, Z., Liao, C., Hang, X., Li, L., Delahaye, D., & Hansen, M. H. (2022) · Sustainability · doi:10.3390/su142215180 · Cited by 44
Automatic detection of urban flood level with YOLOv8 using flooded vehicle dataset
Wan, J., Qin, Y., Shen, Y., Yang, T., Xu, Y., Zhang, S., Yang, G., Xue, F., & Wang, Q. J. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131625 · Cited by 42
Machine learning-based prediction of seismic response of elevated steel tanks
Naeim, B., Akbarzadeh, M. R., & Jahangiri, V. (2024) · Structures · doi:10.1016/j.istruc.2024.107649 · Cited by 35
Evaluation of machine learning applied in membrane-based water desalination: A review
Zhang, Y., He, J., Li, D., Wang, J., Song, Q., Wang, X., Zhu, J., Feng, X., Duan, K., Gao, X., et al. (2025) · Desalination · doi:10.1016/j.desal.2025.119041 · Cited by 41
Meng, Q., Gao, Y., He, Y., Hussain, S., Lu, J., Chen, Y., Qin, C., Luo, F., & Guerrero, J. M. (2026) · Applied Energy · doi:10.1016/j.apenergy.2026.127625 · Cited by 35
BACS: blockchain and AutoML-based technology for efficient credit scoring classification
Yang, F., Qiao, Y., Qi, Y., Bo, J., & Wang, X. (2022) · Annals of Operations Research · doi:10.1007/s10479-022-04531-8 · Cited by 46
“…number of experiments using different learning techniques with a large dataset have demonstrated that 10-fold is an appropriate choice for obtaining the best error estimates while ensuring efficient training as well as not causing redundant time and space wastage (Fushiki 2011; Wong and Yeh 2019).”
Metadata
- Authors
- Wong, T.-T., & Yeh, P.-Y.
- Affiliations
- National Cheng Kung University (TW)
- ORCID
- Wong, Tzu-Tsung
- Venue
- IEEE Transactions on Knowledge and Data Engineering
- Published
- 2020-08-01
- Volume
- 32
- Issue
- 8
- Pages
- 1586-1594
- ISSN
- 1041-4347 (ISSN-L)
- 1558-2191
- 2326-3865
- OpenAlex
- W2942137712
- Cited by
- 1,074
- References
- 18
- Citations by year
- 2026: 178 · 2025: 292 · 2024: 248 · 2023: 178 · 2022: 102 · 2021: 56 · 2020: 18
- FWCI
- 20.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Knowledge and Data Engineering (publishedVersion)
- Funders
- Ministry of Science and Technology, Taiwan (106-2410-H-006-020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Keywords
- k-fold cross-validation, accuracy estimation, repeated cross-validation, k-nearest neighbors, classifier evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.