CoChat

Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial

Open access
Wilimitis, Drew, Walsh, Colin G · 2023 · JMIR AI

Published version of a preprint

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.

Abstract

Cross-validation remains a popular means of developing and validating artificial intelligence for health care. Numerous subtypes of cross-validation exist. Although tutorials on this validation strategy have been published and some with applied examples, we present here a practical tutorial comparing multiple forms of cross-validation using a widely accessible, real-world electronic health care data set: Medical Information Mart for Intensive Care-III (MIMIC-III). This tutorial explored methods such as K-fold cross-validation and nested cross-validation, highlighting their advantages and disadvantages across 2 common predictive modeling use cases: classification (mortality) and regression (length of stay). We aimed to provide readers with reproducible notebooks and best practices for modeling with electronic health care data. We also described sets of useful recommendations as we demonstrated that nested cross-validation reduces optimistic bias but comes with additional computational challenges. This tutorial might improve the community’s understanding of these important methods while catalyzing the modeling community to apply these guides directly in their work using the published code.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/49023

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Pseudocode for nested cross-validation algorithm with model tuning.
Figure 1 Pseudocode for nested cross-validation algorithm with model tuning.CC BY · PubMed Central
Figure 2 Diagram of the methodology for cross-validation (CV) optimism error estimation experiment.
Figure 2 Diagram of the methodology for cross-validation (CV) optimism error estimation experiment.CC BY · PubMed Central
Figure 3 Discrimination for in-hospital mortality prediction by cross-validation (CV) method (with 5 folds used for each method). AUPR: area under the precision-recall curve; AUROC: area under the receiver operator characteristic curve.
Figure 3 Discrimination for in-hospital mortality prediction by cross-validation (CV) method (with 5 folds used for each method). AUPR: area under the precision-recall curve; AUROC: area under the receiver operator characteristic curve.CC BY · PubMed Central
Figure 4 Cross-validation (CV) estimates versus validation set performance (AUROC) for in-hospital mortality by number of folds. AUROC: area under the receiver operator characteristic curve.
Figure 4 Cross-validation (CV) estimates versus validation set performance (AUROC) for in-hospital mortality by number of folds. AUROC: area under the receiver operator characteristic curve.CC BY · PubMed Central
Figure 5 Cross-validation (CV) estimates versus validation set performance (AUPR) for in-hospital mortality by number of folds. AUPR: area under the precision-recall curve.
Figure 5 Cross-validation (CV) estimates versus validation set performance (AUPR) for in-hospital mortality by number of folds. AUPR: area under the precision-recall curve.CC BY · PubMed Central
Figure 6 Cross-validation (CV) estimates versus validation set performance (AUROC) for in-hospital mortality over repeated 5-fold trials. AUROC: area under the receiver operator characteristic curve.
Figure 6 Cross-validation (CV) estimates versus validation set performance (AUROC) for in-hospital mortality over repeated 5-fold trials. AUROC: area under the receiver operator characteristic curve.CC BY · PubMed Central
Figure 7 Cross-validation (CV) estimates versus validation set performance (AUPR) for in-hospital mortality over repeated 5-fold trials. AUPR: area under the precision-recall curve.
Figure 7 Cross-validation (CV) estimates versus validation set performance (AUPR) for in-hospital mortality over repeated 5-fold trials. AUPR: area under the precision-recall curve.CC BY · PubMed Central
Figure 8 Computational time required by cross-validation (CV) method and number of folds for in-hospital mortality prediction. CV: cross-validation.
Figure 8 Computational time required by cross-validation (CV) method and number of folds for in-hospital mortality prediction. CV: cross-validation.CC BY · PubMed Central
Figure 9 Regression metrics by cross-validation (CV) method for length of stay prediction (with 5 folds used for each method).
Figure 9 Regression metrics by cross-validation (CV) method for length of stay prediction (with 5 folds used for each method).CC BY · PubMed Central
Figure 10 Cross-validation (CV) estimates versus validation set performance (Median Absolute Error) by number of folds for length of stay prediction.
Figure 10 Cross-validation (CV) estimates versus validation set performance (Median Absolute Error) by number of folds for length of stay prediction.CC BY · PubMed Central
Figure 11 Cross-validation (CV) estimates versus validation set performance (Mean Absolute Error) by number of folds for length of stay prediction.
Figure 11 Cross-validation (CV) estimates versus validation set performance (Mean Absolute Error) by number of folds for length of stay prediction.CC BY · PubMed Central
Figure 12 Computational time required by cross-validation (CV) method and number of folds for length of stay prediction. CV: cross-validation.
Figure 12 Computational time required by cross-validation (CV) method and number of folds for length of stay prediction. CV: cross-validation.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 33 references

Research works printed but not linked: 3

  1. Mosteller F Tukey JW Lindzey G Aronson E Data analysis, including statistics Handbook of Social Psychology, Volume 2 1968 Boston, MA Addison-Wesley 1 26
  2. Kohavi R A study of cross-validation and bootstrap for accuracy estimation and model selection Proceedings of the 14th International Joint Conference on Artificial Intelligence - Volume 2 1995 IJCAI'95 August 20-25, 1995 Montreal, Quebec
  3. Cawley GC Talbot NL On over-fitting in model selection and subsequent selection bias in performance evaluation J Mach Learn Res 2010 11 2079 107

Cited by

Items 101–186 of 186

Metadata

Authors
Wilimitis, D., & Walsh, C. G.
Affiliations
Vanderbilt University (US)
Corresponding author
Walsh, Colin G
Venue
JMIR AI
Published
2023-12-18
Volume
2
Pages
e49023
ISSN
2817-1705 (ISSN-L)
OpenAlex
W4389883512
Cited by
185
References
33
Citations by year
2026: 113 · 2025: 63 · 2024: 9
FWCI
17.44
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR AI (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • National Institutes of Health
  • National Institute of Mental Health (R01MH120122, R01MH118233, R01MH121455, R01MH116269)
  • National Human Genome Research Institute (RM1HG009034, U54HG012510)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
cross-validation, nested cross-validation, k-fold cross-validation, model validation, artificial intelligence in healthcare, MIMIC-III, mortality prediction, length of stay prediction, electronic health records, optimism bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.