Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial
Open accessPublished version of a preprint
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Cross-validation remains a popular means of developing and validating artificial intelligence for health care. Numerous subtypes of cross-validation exist. Although tutorials on this validation strategy have been published and some with applied examples, we present here a practical tutorial comparing multiple forms of cross-validation using a widely accessible, real-world electronic health care data set: Medical Information Mart for Intensive Care-III (MIMIC-III). This tutorial explored methods such as K-fold cross-validation and nested cross-validation, highlighting their advantages and disadvantages across 2 common predictive modeling use cases: classification (mortality) and regression (length of stay). We aimed to provide readers with reproducible notebooks and best practices for modeling with electronic health care data. We also described sets of useful recommendations as we demonstrated that nested cross-validation reduces optimistic bias but comes with additional computational challenges. This tutorial might improve the community’s understanding of these important methods while catalyzing the modeling community to apply these guides directly in their work using the published code.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Stratified cross-validation can assist in establishing representative class proportions within each fold to improve model generalizability64.”
From Bias recognition and mitigation strategies in artificial intelligence healthcare applications (2025) · cites this paper as [64]
In the citing paper
“This technique is reliant on the availability of large data resources and may not fully account for minority subgroups when not sufficiently represented.”
From Bias recognition and mitigation strategies in artificial intelligence healthcare applications (2025) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · cited 1×
“…this should mitigate the source of optimistic bias introduced when cross-validation is used to tune model parameters on the same data used for model performance evaluation (ie, observed performance can be spuriously high owing to randomness in the data and the learning algorithm) [28,29].”
MethodAssessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Antlanger, M., Aschauer, S., Kopecky, C., Hecking, M., Kovarik, J. J., Werzowa, J., Mascherbauer, J., Genser, B., Säemann, M. D., & Bonderman, D. (2017) · Kidney and Blood Pressure Research · doi:10.1159/000473868 · Cited by 2,808
Bias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485 · cited 1×
“…this should mitigate the source of optimistic bias introduced when cross-validation is used to tune model parameters on the same data used for model performance evaluation (ie, observed performance can be spuriously high owing to randomness in the data and the learning algorithm) [28,29].”
MethodEstimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300 · cited 1×
“Traditional bootstrapping is referred to as out-of-bag bootstrapping, whereas further improvements include the “0.632” and “0.632+” methods that apply additional forms of bias adjustment [23,24].”
MethodInfluentialTowards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
Improvements on Cross-Validation: The 632+ Bootstrap Method
Efron, B., & Tibshirani, R. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474007 · Cited by 1,636 · cited 1×
“Traditional bootstrapping is referred to as out-of-bag bootstrapping, whereas further improvements include the “0.632” and “0.632+” methods that apply additional forms of bias adjustment [23,24].”
MethodInfluentialSteyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · cited 1×
“Clinical usefulness based in decision analysis is the third major area of evaluation [22].”
BackgroundSelection bias in gene extraction on the basis of microarray gene-expression data
Ambroise, C., & McLachlan, G. J. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.102102699 · Cited by 1,480 · cited 1×
“Cross-validation surveys with practical examples, such as those involving microarray and neurologic data, have been published [12,13].”
Background
Research works printed but not linked: 3
- Mosteller, F, 1968, Handbook of Social Psychology, Volume 2, 1
- KohaviRA study of cross-validation and bootstrap for accuracy estimation and model selectionProceedings of the 14th International Joint Conference on Artificial Intelligence - Volume 21995IJCAI'95August 20-25, 1995Montreal, Quebec
- Cawley, GC, 2010, J Mach Learn Res, 11, 2079
Cited by
Bias recognition and mitigation strategies in artificial intelligence healthcare applications
Hasanzadeh, F., Josephson, C. B., Waters, G., Adedinsewo, D., Azizi, Z., & White, J. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01503-7 · Cited by 320
“Strati fi ed cross-validation can assist in establishing representative class proportions within each fold to improve model generalizability 64 .”
Lumumba, V. W., Kiprotich, D., Mpaine, M. L., Makena, N. G., & Kavita, M. D. (2024) · American Journal of Theoretical and Applied Statistics · doi:10.11648/j.ajtas.20241305.13 · Cited by 238
Kasartzian, D.-I., & Tsiampalis, T. (2025) · Life · doi:10.3390/life15010094 · Cited by 73
“While cross-validation can mitigate some imbalances within datasets, it often fails to capture the full range of real-world clinical scenarios [39].”
Deng, J., Elghobashy, M. E., Zang, K., Patel, S. K., Guo, E., & Heybati, K. (2025) · Medical Decision Making · doi:10.1177/0272989x251343082 · Cited by 40
“46”
Ji, H. M., Zhang, X., Wang, T., Yang, K., Jiang, J., & Xing, Z. (2025) · Process Safety and Environmental Protection · doi:10.1016/j.psep.2025.107264 · Cited by 29
Metadata
- Authors
- Wilimitis, D., & Walsh, C. G.
- Affiliations
- Vanderbilt University (US)
- Corresponding author
- Walsh, Colin G
- Venue
- JMIR AI
- Publisher
- JMIR Publications Inc.
- Published
- 2023-12-18
- Volume
- 2
- Pages
- e49023
- ISSN
- 2817-1705 (ISSN-L)
- PMC
- PMC11041453
- OpenAlex
- W4389883512
- Cited by
- 185
- References
- 33
- Citations by year
- 2026: 113 · 2025: 63 · 2024: 9
- FWCI
- 17.44
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- JMIR AI (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health
- National Institute of Mental Health (R01MH120122, R01MH118233, R01MH121455, R01MH116269)
- National Human Genome Research Institute (RM1HG009034, U54HG012510)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- cross-validation, nested cross-validation, k-fold cross-validation, model validation, artificial intelligence in healthcare, MIMIC-III, mortality prediction, length of stay prediction, electronic health records, optimism bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial. https://app.cochat.ai/papers/10.2196/49023
- The paper
- Wilimitis, D., & Walsh, C. G. (2023). Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial. JMIR AI. https://doi.org/10.2196/49023
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.