Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial
Open accessPublished version of a preprint
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Cross-validation remains a popular means of developing and validating artificial intelligence for health care. Numerous subtypes of cross-validation exist. Although tutorials on this validation strategy have been published and some with applied examples, we present here a practical tutorial comparing multiple forms of cross-validation using a widely accessible, real-world electronic health care data set: Medical Information Mart for Intensive Care-III (MIMIC-III). This tutorial explored methods such as K-fold cross-validation and nested cross-validation, highlighting their advantages and disadvantages across 2 common predictive modeling use cases: classification (mortality) and regression (length of stay). We aimed to provide readers with reproducible notebooks and best practices for modeling with electronic health care data. We also described sets of useful recommendations as we demonstrated that nested cross-validation reduces optimistic bias but comes with additional computational challenges. This tutorial might improve the community’s understanding of these important methods while catalyzing the modeling community to apply these guides directly in their work using the published code.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Stratified cross-validation can assist in establishing representative class proportions within each fold to improve model generalizability64.”
From Bias recognition and mitigation strategies in artificial intelligence healthcare applications (2025) · cites this paper as [64]
In the citing paper
“This technique is reliant on the availability of large data resources and may not fully account for minority subgroups when not sufficiently represented.”
From Bias recognition and mitigation strategies in artificial intelligence healthcare applications (2025) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · cited 1×
“…this should mitigate the source of optimistic bias introduced when cross-validation is used to tune model parameters on the same data used for model performance evaluation (ie, observed performance can be spuriously high owing to randomness in the data and the learning algorithm) [28,29].”
MethodAssessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Antlanger, M., Aschauer, S., Kopecky, C., Hecking, M., Kovarik, J. J., Werzowa, J., Mascherbauer, J., Genser, B., Säemann, M. D., & Bonderman, D. (2017) · Kidney and Blood Pressure Research · doi:10.1159/000473868 · Cited by 2,808
Bias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485 · cited 1×
“…this should mitigate the source of optimistic bias introduced when cross-validation is used to tune model parameters on the same data used for model performance evaluation (ie, observed performance can be spuriously high owing to randomness in the data and the learning algorithm) [28,29].”
MethodEstimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300 · cited 1×
“Traditional bootstrapping is referred to as out-of-bag bootstrapping, whereas further improvements include the “0.632” and “0.632+” methods that apply additional forms of bias adjustment [23,24].”
MethodInfluentialTowards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
Improvements on Cross-Validation: The 632+ Bootstrap Method
Efron, B., & Tibshirani, R. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474007 · Cited by 1,636 · cited 1×
“Traditional bootstrapping is referred to as out-of-bag bootstrapping, whereas further improvements include the “0.632” and “0.632+” methods that apply additional forms of bias adjustment [23,24].”
MethodInfluentialSteyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · cited 1×
“Clinical usefulness based in decision analysis is the third major area of evaluation [22].”
BackgroundSelection bias in gene extraction on the basis of microarray gene-expression data
Ambroise, C., & McLachlan, G. J. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.102102699 · Cited by 1,480 · cited 1×
“Cross-validation surveys with practical examples, such as those involving microarray and neurologic data, have been published [12,13].”
Background
Research works printed but not linked: 3
- Mosteller F Tukey JW Lindzey G Aronson E Data analysis, including statistics Handbook of Social Psychology, Volume 2 1968 Boston, MA Addison-Wesley 1 26
- Kohavi R A study of cross-validation and bootstrap for accuracy estimation and model selection Proceedings of the 14th International Joint Conference on Artificial Intelligence - Volume 2 1995 IJCAI'95 August 20-25, 1995 Montreal, Quebec
- Cawley GC Talbot NL On over-fitting in model selection and subsequent selection bias in performance evaluation J Mach Learn Res 2010 11 2079 107
Cited by
Bias recognition and mitigation strategies in artificial intelligence healthcare applications
Hasanzadeh, F., Josephson, C. B., Waters, G., Adedinsewo, D., Azizi, Z., & White, J. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01503-7 · Cited by 320
“Strati fi ed cross-validation can assist in establishing representative class proportions within each fold to improve model generalizability 64 .”
Lumumba, V. W., Kiprotich, D., Mpaine, M. L., Makena, N. G., & Kavita, M. D. (2024) · American Journal of Theoretical and Applied Statistics · doi:10.11648/j.ajtas.20241305.13 · Cited by 243
Kasartzian, D.-I., & Tsiampalis, T. (2025) · Life · doi:10.3390/life15010094 · Cited by 73
“While cross-validation can mitigate some imbalances within datasets, it often fails to capture the full range of real-world clinical scenarios [39].”
Deng, J., Elghobashy, M. E., Zang, K., Patel, S. K., Guo, E., & Heybati, K. (2025) · Medical Decision Making · doi:10.1177/0272989x251343082 · Cited by 40
“46”
Ji, H. M., Zhang, X., Wang, T., Yang, K., Jiang, J., & Xing, Z. (2025) · Process Safety and Environmental Protection · doi:10.1016/j.psep.2025.107264 · Cited by 29
Amanollahi, M., Jameie, M., Looha, M. A., Basti, F. A., Cattarinussi, G., Moghaddam, H. S., Camillo, F. D., Akhondzadeh, S., Pigoni, A., Sambataro, F., et al. (2024) · Journal of Affective Disorders · doi:10.1016/j.jad.2024.06.061 · Cited by 27
Carstens, M., Vasisht, S., Zhang, Z., Barbur, I., Reinke, A., Maier-Hein, L., Hashimoto, D. A., & Kolbinger, F. R. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02227-4 · Cited by 49
“2).”
Jafari, Z., Andrew, M. K., & Rockwood, K. J. (2025) · Age and Ageing · doi:10.1093/ageing/afaf316 · Cited by 32
“Ensemble methods, including bagging and boosting, mitigate overfitting by combining multiple models, which decreases variance, increases robustness and improves generalisation [ 81, 82].”
A scoping review of automatic and semi-automatic MRI segmentation in human brain imaging
Chau, M. T., Vu, H., Debnath, T., & Rahman, M. G. (2025) · Radiography · doi:10.1016/j.radi.2025.01.013 · Cited by 24
Participant-Aware Model Validation for Repeated-Measures Data: Comparative Cross-Validation Study
Karbalaie, A., Abtahi, F., & Häger, C. K. (2026) · JMIR AI · doi:10.2196/87728 · Cited by 27
“…repeated cycles in prediction 5-fold across cycles Genomic prediction Islam et al [38] No computational benchmarking or model ranking Tutorial on subject-aware evaluation Group CV and LOPOCV Clinical machine learning Wilimitis and Walsh [22] No subject-level CV or overfitting assessment”
Consolidated Reporting Guidelines for Prognostic and Diagnostic Machine Learning Models (CREMLS)
El Emam, K., Leung, T. I., Malin, B., Klement, W., & Eysenbach, G. (2024) · Journal of Medical Internet Research · doi:10.2196/52508 · Cited by 16
A Novel Machine and Deep Learning–Based Ensemble Techniques for Automatic Lung Cancer Detection
Ansari, M. M., Kumar, S., Chola, C., Heyat, M. B. B., Akhtar, F., Hayat, M. A. B., Sayeed, E., Parveen, S., Abbasi, R., & Pomary, D. (2025) · BioMed Research International · doi:10.1155/bmri/6666688 · Cited by 22
“Furthermore, AUC results are averaged over fi vefold cross-validation to ensure robustness and generalizability [97].”
Merler, M., Agurto, C., Peller, J., Roitberg, E., Taitz, A., Trevisan, M. A., Navar, I., Berry, J. D., Fraenkel, E., Ostrow, L. W., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01654-7 · Cited by 22
“Followingstandardpracticeinmachinelearningfordatasetsthesizeof ours (1798 examples, 124 subjects) 61,62 , performance evaluation was conducted through fi ve-fold and ten-fold cross-validation, with splits performed using a strati fi ed group k-fold, which separated subjects and sessions.”
Machine Learning-Driven prognosis of tensile strength in FDM printed PLA at various raster angles
Nitin Chandra, A., Bogu, V. P., Jatavallabhula, J. K., Gundeti, S. R., Pappula, B., & Makgato, S. S. (2025) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-025-17127-w · Cited by 13
“Core scientific computing relied on NumPy and Pan-das for array/table operations, while the learning stack used Scikit-learn for preprocessing, pipelines, CV, and evalu - ation, together with the official XGBoost Python API for gradient-boosted trees [33].”
Araujo, M. L. D., Winger, T., Ghosn, S., Saab, C. Y., Srivastava, J., Kazaglis, L., Mathur, P., & Mehra, R. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.04.033 · Cited by 18
“At a minimum, internal validation with clearly documented cross-validation techniques (e.g., stratified k-fold) should be reported [37].”
Predicting Thoracic Aortic Dissection in a Diverse Biobank Using a Polygenic Risk Score
DePaolo, J. S., Zamirpour, S., Abramowitz, S. A., Biagetti, G. M., Judy, R., Beeche, C., Duda, J., Gee, J. C., Witschey, W. R. T., Chirinos, J. A., et al. (2025) · JACC Advances · doi:10.1016/j.jacadv.2025.101743 · Cited by 15
Deng, J., Heybati, K., & Yadav, H. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-092594 · Cited by 14
“A nested CV methodology 12 will be used to evaluate the performance and consistency of the overall ML modelling process.”
Diagnostic accuracy of AI in chest radiography for pneumonia and lung cancer: A meta-analysis
Essa, M. E. A. (2025) · European Journal of Radiology Open · doi:10.1016/j.ejro.2025.100701 · Cited by 15
Rao, A. P., Ranjan, R., Sahana, B. C., & Kumar, G. P. (2025) · Physical and Engineering Sciences in Medicine · doi:10.1007/s13246-024-01512-y · Cited by 11
The role of machine learning in autism spectrum disorder assessment and management
Reilly, A., Walsh, N., O’Reilly, D. P., Smyth, M., Gorman, K. M., Ostadabbas, S., & Power, C. (2025) · Pediatric Research · doi:10.1038/s41390-025-04566-0 · Cited by 18
AIpollen: An Analytic Website for Pollen Identification Through Convolutional Neural Networks
Yu, X., Zhao, J., Xu, Z., Wei, J.-R., Wang, Q., Shen, F., Yang, X., & Guo, Z. (2024) · Plants · doi:10.3390/plants13223118 · Cited by 10
“To further ensure the robustness of our model and to check for any potential biases, we conducted a 5-fold cross-validation on the validation set [40].”
Carstens, M., Vasisht, S., Zhang, Z., Barbur, I., Reinke, A., Maier‐Hein, L., Hashimoto, D. A., & Kolbinger, F. R. (2025) · medRxiv · doi:10.1101/2025.07.12.25330122 · Cited by 9
“Cross-validation, although more computationally intensive, reduces this bias and improves generalizability, which is particularly important for the small datasets that are common in SSU research 31,32 .”
Zhou, F., Parrish, R., Afzal, M., Saha, A., Haynes, R. B., Iorio, A., & Lokker, C. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104825 · Cited by 9
Quantifying training response in cycling based on cardiovascular drift using machine learning
Barsumyan, A., Shyla, R., Saukkonen, A., Soost, C., Graw, J. A., & Burchard, R. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1623384 · Cited by 12
“The final performance of the model was assessed based on its accuracy in predicting the test set, with the results of both the cross-validation and the unseen test set being used to compare the models’ predictive power (Wilimitis and Walsh, 2023).”
Zoha, S., Alam, S., Sifat, I. K., Sultana, N., & Kibria, M. K. (2025) · PLOS Global Public Health · doi:10.1371/journal.pgph.0005494 · Cited by 10
Machine learning predictions of tumor progression: How reliable are we?
Sakkal, M., & Hajal, A. A. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110156 · Cited by 8
Predicting anorexia nervosa treatment efficacy: an explainable machine learning approach
Brizzi, G., Pupillo, C., Sajno, E., Boltri, M., Brusa, F., Scarpina, F., Mendolicchio, L., & Riva, G. (2025) · Journal of Eating Disorders · doi:10.1186/s40337-025-01265-3 · Cited by 9
“…the most suitable cross-validation types for small clinical datasets with multi-characteristic subjects, ensuring a balanced trade-off between bias (errors from overly simplistic models that do not fit the data) and variance (errors from excessively complex models that overfit the data) [45, 46].”
Ma, R., Zu, C., Li, H., Xin, L., Liang, R., Miao, C., Wang, Y., Zhu, Z., & Cui, P. (2025) · ACS Sustainable Chemistry & Engineering · doi:10.1021/acssuschemeng.5c10092 · Cited by 6
Machine Learning Model to Predict Postmastectomy Breast Reconstruction Complications
Shaheen, M. S., McManus, B. T., Cullen, C. M., Chen, J., Momeni, A., Kuo, C., Tsai, P.-H., & Chung, K. C. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.7232 · Cited by 8
3D multiparametric ultrasound imaging of steatotic liver disease in a study with male rats
Lee, D., Heo, J., Mun, H., Oh, D., Ahn, Y. J., & Kim, C. (2025) · Nature Communications · doi:10.1038/s41467-025-65046-x · Cited by 8
“NestedMonteCarlocross-validationwasemployedtoidentifythe optimal feature subset and assess the model ’ s generalization performance while minimizing the riskof over fi tting 87,88 .”
Benachour, Y., Maloukh, L., Bouamama, S., & Geusens, B. (2025) · IEEE Access · doi:10.1109/access.2025.3641629 · Cited by 8
“Coupled with rigorous validation – ensemble methods boosting model robustness [66] and k-fold cross-validation ensuring less biased generalization [67] – these results reinforce the model’s reliability and translational relevance in dermatology.”
KİREMİT, B. Y., & Şahin, D. Ö. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110825 · Cited by 5
Araújo, M., Winger, T., Ghosn, S., Saab, C. Y., Srivastava, J., Kazaglis, L., Mathur, P., & Mehra, R. (2025) · medRxiv · doi:10.1101/2025.02.27.25322950 · Cited by 5
Polygenic Scoring for Detection of Ascending Thoracic Aortic Dilation
DePaolo, J. S., Biagetti, G. M., Judy, R., Wang, G. J., Kelly, J. J., Iyengar, A., Goel, N. J., Desai, N. D., Szeto, W. Y., Bavaria, J. E., et al. (2024) · Circulation Genomic and Precision Medicine · doi:10.1161/circgen.123.004512 · Cited by 5
Exploring Adverse Event Associations of Predicted PXR Agonists Using the FAERS Database
Yamada, S., & Uesawa, Y. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms26157630 · Cited by 4
“To evaluate the model’s performance, we applied nested cross-validation [40], a validation method equivalent to external validation, to avoid overfitting and accurately evaluate the generalization performance.”
Goh, H., Heo, T., Kim, J., Kim, Y., Lim, B., Woo, M. K., Choi, K. Y., Kim, C., & Kim, J. (2025) · Scientific Reports · doi:10.1038/s41598-025-32334-x · Cited by 5
Tavares-Cohén, G. A., Castro, A. P. S., Sales, C. A., Cavalcante, P. A. D. M., de Sena, M. P. M., Alves, B. C. B., El-Awar, U. G., Araújo-Sardinha, A. W., Ribeiro, C. H. M. A., Vieira, J. L. F., et al. (2025) · Brazilian Journal of Biology · doi:10.1590/1519-6984.293623 · Cited by 5
“Incidence projections were carried out using multivariate predictive modeling with cross-validation to enhance the robustness of the estimates, following the recommendations of Wilimitis and Walsh (2023).”
Ciortea, D.-A., Petrea, C. L., Verga, G. I., Berbece, S. I., Gurău, G., Fotea, S., & Matei, M. N. (2025) · Biomedicines · doi:10.3390/biomedicines13102573 · Cited by 4
“Similar pediatric ML studies—whether in cardiology, nephrology, or neurodevelopmental disorders—have demonstrated that internal validation is a meaningful step prior to multicenter confirmation [42,64,68,69,73–75].”
Almuniri, I., Alhussian, H., Aziz, N., Khairy, S. O. F., AlAbri, A. S., Jarallah, Z. F., Saidu, Y., & Adamu, S. (2026) · Scientific Reports · doi:10.1038/s41598-025-31215-7 · Cited by 5
“This protocol mitigates variance associated with single-split evaluations and yields statistically consistent estimates across multiple resampled partitions 45 .”
Enhancement Without Contrast: Stability-Aware Multicenter Machine Learning for Glioma MRI Imaging
Amiri, S., Taeb, S., Gharibi, S., Dehghanfard, S., Mehrnia, S. S., Oveisi, M., Hacihaliloglu, I., Rahmim, A., & Salmanpour, M. R. (2026) · Inventions · doi:10.3390/inventions11010011 · Cited by 3
Moufid, J., Koulali, R., Moussaid, K., & Abghour, N. (2026) · Technologies · doi:10.3390/technologies14020115 · Cited by 3
“These properties are generally viewed as compatible with the limited dataset sizes typically encountered in hospital and medical device contexts [29,30].”
Zhao, Y., & Bergmann, J. (2026) · Building and Environment · doi:10.1016/j.buildenv.2026.114368 · Cited by 4
Zhou, E., Zhang, J., Sun, Y., & Wang, S. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.114973 · Cited by 3
Type 1 Diabetes and Incident Dementia: An Analysis in the All of Us Cohort
Pederson, A. M., Buto, P. T., Zimmerman, S. C., Velez, M., Sims, K. D., Murchland, A. R., Wang, J., Brennan, A. T., Glymour, M. M., & Weuve, J. (2026) · Neurology · doi:10.1212/wnl.0000000000214805 · Cited by 4
Alsiyabi, A., Shahid, S. A., & Al‐Harrasi, A. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.22.600223 · Cited by 3
“Due to the limited number of samples, 3-fold cross-validation was deployed as the resampling procedure to evaluate model performance and identify the best ML algorithm [42].”
MethodSakai, Y., Kim, J., Phi, H. Q., Hu, A. C., Balali, P., Guggenberger, K. V., Woo, J. H., Bos, D., Kasner, S. E., Cucchiara, B. L., et al. (2024) · medRxiv · doi:10.1101/2024.10.25.24316081 · Cited by 3
“Group-fold cross-validation [25] based on Plaque ID ensured no calcifications from the same plaque appeared in both training and validation sets, preventing data leakage.”
InfluentialSacco, P., & Jeong, J. (2025) · Addictive Behaviors · doi:10.1016/j.addbeh.2025.108372 · Cited by 2
Espínola-Sánchez, M., Ríos, A. L., Campaña-Acuña, A., & Sanca-Valeriano, S. (2025) · Digital Health · doi:10.1177/20552076251342012 · Cited by 5
“20 Additionally, we utilized fetal biometric measurements that are widely accepted for fetal weight estimation, 5,8 in alignment with the work of Miotto et al. 14 and Helm et al., 21 who emphasize the importance of developing interpretable models.”
Rugji, J., Erol, Z., Hamadani, A., Taşçı, F., & Horasan, H. (2025) · Applied Food Research · doi:10.1016/j.afres.2025.101321 · Cited by 2
Penttala, M. T., Räisänen, I. T., Sakellari, D., Grigoriadis, A., & Sorsa, T. (2025) · Dentistry Journal · doi:10.3390/dj13110508 · Cited by 3
“There are many variations of cross-validation, but the idea is to test the model’s ability to predict new data that was not used in creating the model (i.e., using one part of the sample data for modeling and the rest of the data for testing) [94,95].”
Xiong, H., Xiang, X.-L., Liu, X., Yang, T., Chen, J., Chen, J., & Li, T. (2025) · Child and Adolescent Psychiatry and Mental Health · doi:10.1186/s13034-025-00988-0 · Cited by 3
Adi, S., Bosson-Rieutort, D., Breton, M., Smithman, M. A., Laberge, M., & Sourial, N. (2026) · BMC Health Services Research · doi:10.1186/s12913-025-13941-w · Cited by 2
“We partitioned the dataset into training and test sets (75/25 ratio) and applied five-fold cross-validation within the training set, following standard practice, [48,49] to assess performance.”
Mahwati, Y., Tenrisau, D., Hasibuan, S. R., Wilaksono, B., Indriyani, Y., Abdullah, A. A., Malik, H., & Zainuddin, A. A. (2026) · Journal of Preventive Medicine and Public Health · doi:10.3961/jpmph.25.350 · Cited by 2
“The dataset comprises a 1% representative sample of Indonesia’s National Health Insurance members, selected through stratified random sampling based on facility type and family category [17].”
Interpretable machine learning for bridge-pier scour prediction and flood resilience
Khan, A., & Ismael, D. (2026) · Frontiers in Built Environment · doi:10.3389/fbuil.2025.1731114 · Cited by 7
“In each cycle, one-fold serves as the validation set while the remaining (K-1) folds constitute the temporary training subset (Teodorescu and Obreja Braşoveanu, 2025; Wilimitis and Walsh, 2023).”
Arshad, M., & Emad, M. Z. (2026) · Results in Engineering · doi:10.1016/j.rineng.2026.109430 · Cited by 2
Reddy, B. S., Saroja, Y., Ponugoti, G. R., Kumar, P. P., Jatavallabhula, J. K., Babu, V. V. P., & Pappula, B. (2026) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-026-17524-9 · Cited by 2
Improving Cross-Domain Generalization in Brain MRIs via Feature Space Stability Regularization
Kakon, S. C., Jamwal, H. D. S., & Singh, S. (2026) · Mathematics · doi:10.3390/math14061082 · Cited by 3
“Standard k-fold cross-validation within a single dataset primarily evaluates internal consistency and may yield overly optimistic performance estimates that fail to reflect real-world distributional variability [33].”
Vajrala, A., Reddy, M. R., & Bailey, R. C. (2026) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2026.118842 · Cited by 2
Oguzhanoglu, M. K., Oguzhanoglu, K., Karacabey, S., Atas, B. S., & Çetin, A. (2026) · BMC Endocrine Disorders · doi:10.1186/s12902-026-02284-0 · Cited by 3
“An inner cross-validation loop (5 folds) optimized hyperparameters, while an outer loop (5 folds) assessed model generalization [25].”
Deng, J., Gharavi, A. E., Heybati, K., Haile, D., Ahn, E. S., Mardini, S., & Gibreel, W. O. (2026) · Journal of Plastic Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2026.05.033 · Cited by 2
Gudiño-Ochoa, A., Ruiz-Velazquez, E., García-Rodríguez, J. A., Ochoa-Ornelas, R., & Uribe-Toscano, S. (2026) · AI · doi:10.3390/ai7060213 · Cited by 2
Turky, M., Samaranayake, L. P., Osathanon, T., & Dummer, P. M. H. (2026) · International Dental Journal · doi:10.1016/j.identj.2026.109704 · Cited by 2
“28 Such evaluation strategies are essential to assess the model’s reliability and applicability across diverse populations.”
Buianova, I., Silvestrin, M., Deng, J. D., & Pornpattananangkul, N. (2025) · eLife · doi:10.7554/elife.108109 · Cited by 3
“Here, we used a nested cross-validation scheme with five outer folds and ten inner folds ( Wilimitis and Walsh, 2023 ).”
Buianova, I., Silvestrin, M., Deng, J. D., & Pornpattananangkul, N. (2026) · eLife · doi:10.7554/elife.108109.3 · Cited by 2
Deng, J., Heybati, K., Poudel, K., Xie, G., Zuberi, E. C., Simha, V., & Yadav, H. (2025) · Journal of Intensive Care Medicine · doi:10.1177/08850666251342559 · Cited by 1
“11 For visualization of nested cross-validation performance, we present averaged ROC plots via vertical averaging, 35 averaged sensitivity-speci fi city trade-off curves, as well as averaged calibration curves generated by aligning individual calibration curves to fi ve common bin centers using…”
Madi, N., Sayeed, M. A., Cato, E. T., Creasy-Marrazzo, A., Islam, K., Khabir, M. I. U., Islam, M. T., Khan, Z. H., Bhuiyan, M. T. R., Begum, Y. A., et al. (2026) · The Journal of Infectious Diseases · doi:10.1093/infdis/jiag286 · Cited by 1
Nguyen, D., Lee, S., Synghal, R., Chan, L., Justus, F., Moromisato, M., Shao, T., Wang, C., Kellogg, M., Anwar, B., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01838-1 · Cited by 4
“This adaptability is crucial for maintaining real-world performance in care navigation on the digital channel 14,15 .”
Küçükerdem, H. K., & Öztürk, H. H. (2025) · Sustainability · doi:10.3390/su17219455 · Cited by 1
Guachamín-Chicaiza, C., Villota-González, F. H., & Zúñiga-Grajeda, V. (2025) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2025.100383 · Cited by 3
Hanifi, A., Abedini‐Nassab, R., & Ashtiani, M. N. (2025) · Diagnostics · doi:10.3390/diagnostics15222855 · Cited by 1
“Furthermore Furthermore, our methodology supports robust validation through nested cross-validation, bootstrapping, and permutation testing, which collectively help reduce the risks of overfitting, information leakage, and false-positive patterns [38,39].”
Domingues, K. Z. A., Lazo, R. E. L., do Amaral, L. S., Cobre, A. D. F., Ferreira, L. M., & Pontarolo, R. (2025) · Analytical Science Advances · doi:10.1002/ansa.70056 · Cited by 1
Khan, R. A., Lambert, P. J., Wang, Z., Abbasian, P., Ryner, L. N., Pitz, M. W., & Ashraf, A. (2025) · International Conference on Biomedical Signal and Image Processing (ICBIP) · doi:10.1109/icbip66535.2025.11284424 · Cited by 1
“The importance of addressing class balance is well-established, as it prevents the model from favoring the majority class, which could distort overall classification metrics [24, 3].”
Büber, M., & Yaşar, A. (2025) · Karamanoğlu Mehmetbey Üniversitesi Mühendislik ve Doğa Bilimleri Dergisi · doi:10.55213/kmujens.1817251 · Cited by 1
Age Prediction of Hematoma from Hyperspectral Images Using Convolutional Neural Networks
Keshavarz, A., Bieber, G., Wulff, D., Babian, C., & Lüdtke, S. (2026) · Journal of Imaging · doi:10.3390/jimaging12020078 · Cited by 2
“All data partitioning was performed on a participant basis to prevent information leakage across time points [21].”
Mabodi, R., Mbunge, E., Makaba, T., & Ntshingila, N. (2026) · Applied Sciences · doi:10.3390/app16073489 · Cited by 1
Boundary Conditions for AU-Based Detection of Understanding: A Literary Analysis Study
Lazic, M., & Woodruff, E. (2026) · Electronics · doi:10.3390/electronics15102059 · Cited by 1
Multimodal Prediction of Periodontitis Using Root Exposure in Intraoral Images and Age
Kang, S., Go, H., Kwon, Y.-E., Choi, Y.-H., Park, E. Y., & Kim, E. (2026) · International Dental Journal · doi:10.1016/j.identj.2026.109617 · Cited by 1
“21 However, although the imaging protocol and AI approach enhance reproducibility, broader validation, including exter-nal, multi-ethnic, gender-balanced, and prospective cohorts, is required for generalizability.”
Jiang, T., Zhang, S., & Jiang, D. (2026) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2026.1809332 · Cited by 2
Ewis, A. A., & Nofal, O. M. (2026) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2026.112900 · Cited by 1
A Generative AI Framework for Carbon-Oriented Biomimetic Façade Design in Architecture
Gai, M., Kam, K. J., Flor, J.-F., Chai, C., & Gunasagaran, S. (2026) · Buildings · doi:10.3390/buildings16112082 · Cited by 1
Zhao, J., Dong, Y., Zhang, J., & Wu, K. (2026) · ACS Omega · doi:10.1021/acsomega.5c12379 · Cited by 1
AI-Based Decision Support System for Safe Pediatric Intravenous Drug Dosage Prediction
Ahmed, M., Nafees, M., Maqsood, I., & Khan, M. H. D. (2026) · International Conference on Robotics and Automation in Industry (ICRAI) · doi:10.1109/icrai70912.2026.11551902 · Cited by 1
“However, such deployment must ensure interpretability and the possibility of human override[8].”
Leuprech, A., Balachandran, A., Deen, A. R., Bhat, S. M., Ting, W., Tharmarasa, R., Deen, M. J., & Howlader, M. M. R. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.115359 · Cited by 1
Mncwabe, M., & Paepae, T. (2026) · Informatics · doi:10.3390/informatics13060094 · Cited by 1
“…that balances bias and variance in model performance estimation; comparative analyses confirm that stratified K-fold cross-validation provides reliable performance estimates for both classification and regression tasks while avoiding the computational overhead of leave-one-out approaches [53,54].”
Guo, Y., Jiang, S., Zhu, W., Tan, L., Liu, C., Jia, Y., Zhang, C., & Chin, K. (2026) · PLoS ONE · doi:10.1371/journal.pone.0351334 · Cited by 1
da Costa, R. C., Pimentel, P. O., Pessoa, A. C. P., Bráz, G., & de Almeida, J. D. S. (2025) · doi:10.5753/sbcas.2025.7270 · Cited by 1
Karbalaie, A., Abtahi, F., & Häger, C. K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5352124 · Cited by 1
Mohagheghi, M., Bahadori‐Jahromi, A., & Room, S. (2026) · Encyclopedia · doi:10.3390/encyclopedia6040075 · Cited by 2
“Many studies consider cross-validation and external testing to be more reliable than simple hold-out validation, as these approaches are less affected by data partitioning and better reflect generalisation performance, particularly in challenging domains like building inspections [95,96].”
El Emam, K., Leung, T. I., Malin, B., Klement, W., & Eysenbach, G. (2024) · doi:10.2196/preprints.52508
Alluri, D. S., & Pabon‐Rodriguez, F. M. (2024) · medRxiv · doi:10.1101/2024.12.21.24319488
“Particularly for complex datasets, such as the one used in this study, 10-fold cross-validation enhances the reliability and generalizability of the results, making it a robust validation strategy for machine learning applications (Wilimitis & Walsh, 2023).”
InfluentialWannous, A. K. (2025) · medRxiv · doi:10.1101/2025.03.31.25324938
“Cross-validation is a key data resampling technique in machine learning, essential for evaluating model generalization and minimizing overfitting [12].”
Moodley, Y., Brink, W., Wyk, J. V., Kader, S., Wexner, S. D., Neugut, A. I., & Kiran, R. P. (2025) · JCO Global Oncology · doi:10.1200/go-24-00480
“24 The regression coef fi cients obtained from the risk model were expo-nentiated to provide odds ratios (ORs) and 95% CI for each variable.”
Deep Learning-Based Automated Detection of Oral Leukoplakia in Clinical Imaging
Li, D., Wang, X., Liu, J., & Sun, J. (2025) · Cureus · doi:10.7759/cureus.83368 · Cited by 7
“Model performance was assessed through stratified 10-fold cross-validation [22].”
Beyond the limits of conventional ‘endpoint’ ELISA and rescuing the signal with lag k-ELISA
Katsanovskaja, K., Marchesin, F., Ujetz, J., Ijaz, S., Tedder, R. S., & McClure, M. O. (2025) · Journal of Virological Methods · doi:10.1016/j.jviromet.2025.115231
Buianova, I., Silvestrin, M., Deng, J. D., & Pornpattananangkul, N. (2025) · eLife · doi:10.7554/elife.108109.1
Enhancing fracture detection in wrist radiographs via paired synthetic data generationPreprint
Norris, S. A., Carrion, D., Uribe, S. A., & Badawy, M. K. (2025) · medRxiv · doi:10.1101/2025.09.18.25336124
“…for this variability, such as k -fold cross-validation, provide more statistically robust estimates of model performance but require training k models, making them computationally expensive [13, 14] and often impractical in clinical research settings where resources and time are limited [15–17].”
Jadidi, S. T., Goweder, A. M., & Abdelhamid, M. H. M. (2025) · Research Square · doi:10.21203/rs.3.rs-7687597/v1
Gray, S., Amadon, G., Temkin, N., Darsie, M. E., Giacino, J. T., Corrigan, J. D., Korley, F. K., Whyte, J., Stein, M. B., Manley, G. T., et al. (2025) · Journal of Emergency Medicine · doi:10.1016/j.jemermed.2025.10.008
Onishi, H., Okamoto, T., Niida, Y., & Yamamura, O. (2025) · The Journal of Medical Investigation · doi:10.2152/jmi.72.272
Salmi, M., Atif, D., & Ventura, S. (2025) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa65442.2025.11248013 · Cited by 1
“Wilimitis et al. [29] provided a practical tutorial on cross-validation in healthcare modeling, comparing K-fold, stratified, nested, and bootstrapping techniques using MIMIC-III.”
Metadata
- Authors
- Wilimitis, D., & Walsh, C. G.
- Affiliations
- Vanderbilt University (US)
- Corresponding author
- Walsh, Colin G
- Venue
- JMIR AI
- Publisher
- JMIR Publications Inc.
- Published
- 2023-12-18
- Volume
- 2
- Pages
- e49023
- ISSN
- 2817-1705 (ISSN-L)
- PMC
- PMC11041453
- OpenAlex
- W4389883512
- Cited by
- 185
- References
- 33
- Citations by year
- 2026: 113 · 2025: 63 · 2024: 9
- FWCI
- 17.44
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- JMIR AI (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health
- National Institute of Mental Health (R01MH120122, R01MH118233, R01MH121455, R01MH116269)
- National Human Genome Research Institute (RM1HG009034, U54HG012510)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- cross-validation, nested cross-validation, k-fold cross-validation, model validation, artificial intelligence in healthcare, MIMIC-III, mortality prediction, length of stay prediction, electronic health records, optimism bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.