Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial
Open accessPublished version of a preprint
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Cross-validation remains a popular means of developing and validating artificial intelligence for health care. Numerous subtypes of cross-validation exist. Although tutorials on this validation strategy have been published and some with applied examples, we present here a practical tutorial comparing multiple forms of cross-validation using a widely accessible, real-world electronic health care data set: Medical Information Mart for Intensive Care-III (MIMIC-III). This tutorial explored methods such as K-fold cross-validation and nested cross-validation, highlighting their advantages and disadvantages across 2 common predictive modeling use cases: classification (mortality) and regression (length of stay). We aimed to provide readers with reproducible notebooks and best practices for modeling with electronic health care data. We also described sets of useful recommendations as we demonstrated that nested cross-validation reduces optimistic bias but comes with additional computational challenges. This tutorial might improve the community’s understanding of these important methods while catalyzing the modeling community to apply these guides directly in their work using the published code.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Stratified cross-validation can assist in establishing representative class proportions within each fold to improve model generalizability64.”
From Bias recognition and mitigation strategies in artificial intelligence healthcare applications (2025) · cites this paper as [64]
In the citing paper
“This technique is reliant on the availability of large data resources and may not fully account for minority subgroups when not sufficiently represented.”
From Bias recognition and mitigation strategies in artificial intelligence healthcare applications (2025) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · cited 1×
“…this should mitigate the source of optimistic bias introduced when cross-validation is used to tune model parameters on the same data used for model performance evaluation (ie, observed performance can be spuriously high owing to randomness in the data and the learning algorithm) [28,29].”
MethodAssessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Antlanger, M., Aschauer, S., Kopecky, C., Hecking, M., Kovarik, J. J., Werzowa, J., Mascherbauer, J., Genser, B., Säemann, M. D., & Bonderman, D. (2017) · Kidney and Blood Pressure Research · doi:10.1159/000473868 · Cited by 2,808
Bias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485 · cited 1×
“…this should mitigate the source of optimistic bias introduced when cross-validation is used to tune model parameters on the same data used for model performance evaluation (ie, observed performance can be spuriously high owing to randomness in the data and the learning algorithm) [28,29].”
MethodEstimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300 · cited 1×
“Traditional bootstrapping is referred to as out-of-bag bootstrapping, whereas further improvements include the “0.632” and “0.632+” methods that apply additional forms of bias adjustment [23,24].”
MethodInfluentialTowards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
Improvements on Cross-Validation: The 632+ Bootstrap Method
Efron, B., & Tibshirani, R. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474007 · Cited by 1,636 · cited 1×
“Traditional bootstrapping is referred to as out-of-bag bootstrapping, whereas further improvements include the “0.632” and “0.632+” methods that apply additional forms of bias adjustment [23,24].”
MethodInfluentialSteyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · cited 1×
“Clinical usefulness based in decision analysis is the third major area of evaluation [22].”
BackgroundSelection bias in gene extraction on the basis of microarray gene-expression data
Ambroise, C., & McLachlan, G. J. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.102102699 · Cited by 1,480 · cited 1×
“Cross-validation surveys with practical examples, such as those involving microarray and neurologic data, have been published [12,13].”
BackgroundPrediction error estimation: a comparison of resampling methods
Molinaro, A. M., Simon, R., & Pfeiffer, R. M. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti499 · Cited by 1,339 · cited 1×
“More advanced resampling methods include bootstrap-based cross-validation, Monte Carlo holdout validation (or repeated holdout sampling), and repeated nested cross-validation [25,26].”
MethodPrediction models need appropriate internal, internal-external, and external validation
Steyerberg, E. W., & Harrell, F. E. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.005 · Cited by 1,304 · cited 1×
“A newer concept of “internal-external” validation has also been suggested for studies with multisite data [9].”
BackgroundRisk prediction models: II. External validation, model updating, and impact assessment
Moons, K. G. M., Kengne, A. P., Grobbee, D. E., Royston, P., Vergouwe, Y., Altman, D. G., & Woodward, M. (2012) · Heart · doi:10.1136/heartjnl-2011-301247 · Cited by 1,195 · cited 1×
“Broadly, validation consists of either internal validation, which should be reported alongside model development, or external validation, in which a developed model is tested in an unseen data set in a new setting [7,8].”
MethodCross-validation pitfalls when selecting and assessing regression and classification models
Krstajic, D., Buturovic, L. J., Leahy, D. E., & Thomas, S. (2014) · Journal of Cheminformatics · doi:10.1186/1758-2946-6-10 · Cited by 1,090 · cited 1×
“More advanced resampling methods include bootstrap-based cross-validation, Monte Carlo holdout validation (or repeated holdout sampling), and repeated nested cross-validation [25,26].”
MethodMoons, K. G. M., Kengne, A. P., Woodward, M., Royston, P., Vergouwe, Y., Altman, D. G., & Grobbee, D. E. (2012) · Heart · doi:10.1136/heartjnl-2011-301246 · Cited by 889 · cited 1×
“Broadly, validation consists of either internal validation, which should be reported alongside model development, or external validation, in which a developed model is tested in an unseen data set in a new setting [7,8].”
MethodAssessing and tuning brain decoders: Cross-validation, caveats, and guidelines
Varoquaux, G., Raamana, P. R., Engemann, D. A., Hoyos-Idrobo, A., Schwartz, Y., & Thirion, B. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.10.038 · Cited by 792 · cited 2×
“This also contributed to a relatively lower reduction in optimistic bias when using the theoretically validated approach to reduce optimistic bias resulting from model tuning in cross-validation (nested methods) [12,30,32].”
BackgroundMethodEstimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap
Kim, J.-H. (2009) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2009.04.009 · Cited by 792
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600
Cross-Validation: What Does It Estimate and How Well Does It Do It?
Bates, S., Hastie, T., & Tibshirani, R. (2024) · Journal of the American Statistical Association · doi:10.1080/01621459.2023.2197686 · Cited by 442 · cited 1×
“When performing hyperparameter tuning and other model selection steps within cross-validation, the difference in optimistic bias observed between nested and nonnested cross-validation methods should have been mitigated [31].”
MethodThe need to approximate the use-case in clinical machine learning
Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P. (2017) · GigaScience · doi:10.1093/gigascience/gix019 · Cited by 367 · cited 1×
“In the former example, record-wise cross-validation might be adopted for diagnosis at a given clinical encounter, and subject-wise cross-validation would be favorable for prognosis over time [19].”
MethodCorrecting for Optimistic Prediction in Small Data Sets
Smith, G. C. S., Seaman, S. R., Wood, A. M., Royston, P., & White, I. R. (2014) · American Journal of Epidemiology · doi:10.1093/aje/kwu140 · Cited by 352
The shrinkage of the coefficient of multiple correlation.
Larson, S. C. (1931) · Journal of Educational Psychology · doi:10.1037/h0072400 · Cited by 233 · cited 1×
“Cross-validation originated in the 1930s with K-fold cross-validation, the most common form of cross-validation, described in the 1960s [16,17].”
MethodBootstrapping the out-of-sample predictions for efficient and accurate cross-validation
Tsamardinos, I., Greasidou, E., & Borboudakis, G. (2018) · Machine Learning · doi:10.1007/s10994-018-5714-4 · Cited by 211
On the Dangers of Cross-Validation. An Experimental Evaluation
Rao, R. B., Fung, G., & Rosales, R. (2008) · Proceedings of the 2008 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972788.54 · Cited by 202 · cited 1×
“This also contributed to a relatively lower reduction in optimistic bias when using the theoretically validated approach to reduce optimistic bias resulting from model tuning in cross-validation (nested methods) [12,30,32].”
MethodTougui, I., Jilbab, A., & Mhamdi, J. E. (2021) · Healthcare Informatics Research · doi:10.4258/hir.2021.27.3.189 · Cited by 199 · cited 1×
“…statistical method, many variations of cross-validation exist with respective strengths and weaknesses, distinct use cases for model development and performance estimation that are often misapplied, and domain-specific considerations necessary for effective health care implementation [10,11].”
MethodCollins, G. S., Omar, O., Shanyinde, M., & Yu, L.-M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.06.020 · Cited by 192
Using and understanding cross-validation strategies. Perspectives on Saeb et al.
Little, M. A., Varoquaux, G., Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P. (2017) · GigaScience · doi:10.1093/gigascience/gix020 · Cited by 158 · cited 1×
“Although the technical details and practical guidelines of subject-wise versus record-wise cross-validation are debated, the best approach depends on the specific use case, the number of records, the size of the data set, and the degree of correlation within subject records [18].”
BackgroundEvaluation of machine learning algorithms for health and wellness applications: A tutorial
Tohka, J., & van Gils, M. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104324 · Cited by 131
A framework for the oversight and local deployment of safe and high-quality prediction models
Bedoya, A. D., Economou-Zavlanos, N. J., Goldstein, B. A., Young, A., Jelovsek, J. E., O’Brien, C., Parrish, A. B., Elengold, S., Lytle, K., Balu, S., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac078 · Cited by 119
Action-Informed Artificial Intelligence—Matching the Algorithm to the Problem
Lindsell, C. J., Stead, W. W., & Johnson, K. B. (2020) · JAMA · doi:10.1001/jama.2020.5035 · Cited by 56
Neeman, T. M. (2009) · International Statistical Review · doi:10.1111/j.1751-5823.2009.00085_22.x · Cited by 40
Li, M., & Du, S. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.912151 · Cited by 13 · cited 1×
“It has been studied extensively, given its relative accessibility compared with most health care modeling studies using EHRs, in which privacy or challenges in at-scale deidentification prohibit data sharing with publication [27].”
BackgroundMIMIC-IV Clinical Database Demo
Johnson, A. E. W., Bulgarelli, L., Pollard, T., Horng, S., Celi, L. A., & Mark, R. G. (2022) · PhysioNet · doi:10.13026/ng9m-3n32 · Cited by 13
Research works printed but not linked: 3
- Mosteller F Tukey JW Lindzey G Aronson E Data analysis, including statistics Handbook of Social Psychology, Volume 2 1968 Boston, MA Addison-Wesley 1 26
- Kohavi R A study of cross-validation and bootstrap for accuracy estimation and model selection Proceedings of the 14th International Joint Conference on Artificial Intelligence - Volume 2 1995 IJCAI'95 August 20-25, 1995 Montreal, Quebec
- Cawley GC Talbot NL On over-fitting in model selection and subsequent selection bias in performance evaluation J Mach Learn Res 2010 11 2079 107
Cited by
Bias recognition and mitigation strategies in artificial intelligence healthcare applications
Hasanzadeh, F., Josephson, C. B., Waters, G., Adedinsewo, D., Azizi, Z., & White, J. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01503-7 · Cited by 320
“Strati fi ed cross-validation can assist in establishing representative class proportions within each fold to improve model generalizability 64 .”
Lumumba, V. W., Kiprotich, D., Mpaine, M. L., Makena, N. G., & Kavita, M. D. (2024) · American Journal of Theoretical and Applied Statistics · doi:10.11648/j.ajtas.20241305.13 · Cited by 243
Kasartzian, D.-I., & Tsiampalis, T. (2025) · Life · doi:10.3390/life15010094 · Cited by 73
“While cross-validation can mitigate some imbalances within datasets, it often fails to capture the full range of real-world clinical scenarios [39].”
Deng, J., Elghobashy, M. E., Zang, K., Patel, S. K., Guo, E., & Heybati, K. (2025) · Medical Decision Making · doi:10.1177/0272989x251343082 · Cited by 40
“46”
Ji, H. M., Zhang, X., Wang, T., Yang, K., Jiang, J., & Xing, Z. (2025) · Process Safety and Environmental Protection · doi:10.1016/j.psep.2025.107264 · Cited by 29
Metadata
- Authors
- Wilimitis, D., & Walsh, C. G.
- Affiliations
- Vanderbilt University (US)
- Corresponding author
- Walsh, Colin G
- Venue
- JMIR AI
- Publisher
- JMIR Publications Inc.
- Published
- 2023-12-18
- Volume
- 2
- Pages
- e49023
- ISSN
- 2817-1705 (ISSN-L)
- PMC
- PMC11041453
- OpenAlex
- W4389883512
- Cited by
- 185
- References
- 33
- Citations by year
- 2026: 113 · 2025: 63 · 2024: 9
- FWCI
- 17.44
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- JMIR AI (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health
- National Institute of Mental Health (R01MH120122, R01MH118233, R01MH121455, R01MH116269)
- National Human Genome Research Institute (RM1HG009034, U54HG012510)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- cross-validation, nested cross-validation, k-fold cross-validation, model validation, artificial intelligence in healthcare, MIMIC-III, mortality prediction, length of stay prediction, electronic health records, optimism bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.