CoChat
Sign Up
Cited by 223Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareMachine Learning in Healthcare

Predictive analytics in health care: how can we know it works?

Open access
Van Calster, Ben, Wynants, Laure, Timmerman, Dirk, Steyerberg, Ewout W, Collins, Gary S·2019·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocz130PMC PMC6857503
Open this paper in CoChat — free accountOpen PDF

Abstract

There is increasing awareness that the methodology and findings of research should be transparent. This includes studies using artificial intelligence to develop predictive algorithms that make individualized diagnostic or prognostic risk predictions. We argue that it is paramount to make the algorithm behind any prediction publicly available. This allows independent external validation, assessment of performance heterogeneity across settings and over time, and algorithm refinement or updating. Online calculators and apps may aid uptake if accompanied with sufficient information. For algorithms based on “black box” machine learning methods, software for algorithm implementation is a must. Hiding algorithms for commercial exploitation is unethical, because there is no possibility to assess whether algorithms work as advertised or to monitor when and how algorithms are updated. Journals and funders should demand maximal transparency for publications on predictive algorithms, and clinical guidelines should only recommend publicly available algorithms.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocz130

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Proprietary models are difficult to assess owing to their opaque nature and have been shown to decline in performance over time.

    “This limited information is of concern because proprietary models are difficult to assess owing to their opaque nature and have been shown to decline in performance over time.15,16”

    From External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients (2021) · cites this paper as [15]

    The “Proprietary algorithms” section explains that undisclosed, fee-for-service models can be costly to validate, supporting the concern that they are difficult to assess. The “Algorithm performance is not guaranteed” section says performance may differ over time and should be monitored, but does not show that proprietary models specifically have declined in performance over time. No page numbers are provided in our copy. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 23

  1. [1]Steyerberg EW. Clinical Prediction Models. New York: Springer; 2009.
  2. [2]Shah ND, Steyerberg EW, Kent DM. Big data and predictive analytics: recalibrating expectations. JAMA 2018; 320 1: 27– 8. 29813156
  3. [3]Beam AL, Kohane IS. Big data and machine learning in health care. JAMA 2018; 319 13: 1317– 8. 29532063
  4. [4]Hripcsak G, Albers DJ. Next-generation phenotyping of electronic health records. J Am Med Inform Assoc 2013; 20 1: 117– 21. 22955496 PMC3555337
  5. [8]Lynch CJ, Liston C. New machine-learning technologies for computer-aided diagnosis. Nat Med 2018; 24 9: 1304– 5. 30177823
  6. [9]Riley RD, Ensor J, Snell KI, et al External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ 2016; 353: i3140. 27334381 PMC4916924
  7. [10]Ghassemi M, Naumann T, Schulam P, Beam AL, Ranganath R. Opportunities in machine learning for healthcare. arxiv: 1806.00388 [cs.LG]; 2018. · arXiv
  8. [12]Testa A, Kaijser J, Wynants L, et al Strategies to diagnose ovarian cancer: new evidence from phase 3 of the multicenter international IOTA study. Br J Cancer 2014; 111 4: 680– 8. 24937676 PMC4134495
  9. [14]Chen JH, Alagappan M, Goldstein MK, et al Decaying relevance of clinical data towards future decisions in data-driven inpatient clinical order sets. Int J Med Inform 2017; 102: 71– 9. 28495350 PMC5459355
  10. [15]Levy-Fix G, Gorman SL, Sepulveda JL, Elhadad N. When to re-order laboratory tests? Learning laboratory test shelf-life. J Biomed Inform 2018; 85: 21– 9. 30036675 PMC11073806
  11. [17]van der Ploeg T, Nieboer D, Steyerberg EW. Modern modeling techniques had limited external validity in predicting mortality from traumatic brain injury. J Clin Epidemiol 2016; 78: 83– 9. 26987507
  12. [18]Nevin L; on behalf of the PLoS Medicine Editors. Advancing the beneficial use of machine learning in health care and medicine: toward a community understanding. PLoS Med 2018; 15 11: e1002708. 30500811 PMC6267950
  13. [19]Goldstein BA, Navar AM, Pencina MJ. Risk prediction with electronic health records. The importance of model validation and clinical context. JAMA Cardiol 2016; 1 9: 976. 27732706 PMC5510620
  14. [21]Moons KGM, Altman DG, Reitsma JB, et al Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015; 162 1: W1– 73. 25560730
  15. [22]Glasziou P, Altman DG, Bossuyt P, et al Reducing waste from incomplete and unusable reports of biomedical research. Lancet 2014; 383 9913: 267– 76. 24411647
  16. [23]Kleinrouweler CE, Cheong-See FM, Collins GS, et al Prognostic models in obstetrics: available, but far from applicable. Am J Obstet Gynecol 2016; 214 1: 79– 90. 26070707
  17. [24]Collins GS, Michaëlsson K. Fracture risk assessment: state of the art, methodologically unsound, or poorly reported. Curr Osteoporos Rep 2012; 10 3: 199– 207. 22688862
  18. [25]Ohno-Machado L. Data science and artificial intelligence to improve clinical practice and research. J Am Med Inform Assoc 2018; 25 10: 1273. 30312446 PMC7646927
  19. [27]Reps JM, Schuemie MJ, Suchard MA, Ryan PB, Rijnbeek PR. Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. J Am Med Inform Assoc 2018; 25 8: 969– 75. 29718407 PMC6077830
  20. [28]Wiegand T, Krishnamurthy R, Kuglitsch M, et al WHO and ITU establish benchmarking process for artificial intelligence in health. Lancet 2019; 394 10192: 9. 30935732
  21. [30]Park SH. Regulatory approval versus clinical validation of artificial intelligence diagnostic tools. Radiology 2018; 288 3: 910– 1. 30040041
  22. [32]Hripcsak G, Ryan PB, Duke JD, et al Characterizing treatment pathways at scale using the OHDSI network. Proc Natl Acad Sci USA 2016; 113 27: 7329– 36. 27274072 PMC4941483
  23. [33]Black L, Knoppers BM, Avard D, et al Legal liability and the uncertain nature of risk prediction: the case of breast cancer risk prediction models. Public Health Genomics 2012; 15 6: 335– 40. 22987123

Cited by

Sorted by citations

Authors
Van Calster, B., Wynants, L., Timmerman, D., Steyerberg, E. W., & Collins, G. S.
Affiliations
  • Department of Development and Regeneration, KU Leuven, Leuven, Belgium
  • KU Leuven (BE)
  • Universitair Ziekenhuis Leuven (BE)
  • Leiden University Medical Center (NL)
  • Centre for Statistics in Medicine, Nuffield, Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, UK
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2019-08-02
Volume
26
Issue
12
Pages
1651-1654
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocz130
PMC
PMC6857503
OpenAlex
W2965782433
Cited by
223
References
10
Citations by year
2026: 30 · 2025: 41 · 2024: 43 · 2023: 32 · 2022: 28 · 2021: 22 · 2020: 26 · 2019: 1
FWCI
6.11
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, cc-by-nc-nd)
  • Lirias (KU Leuven) (submittedVersion, other-oa)
  • Oxford University Research Archive (ORA) (University of Oxford) (submittedVersion, cc-by-nc-nd)
  • University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
  • Research Publications (Maastricht University) (submittedVersion, other-oa)
Funders
Research Foundation
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
predictive analytics, algorithmic transparency, external validation, algorithmic bias, black-box machine learning, algorithmic ethics, research transparency, clinical guidelines, online calculator
MeSH terms
Machine Learning, Algorithms, Forecasting, Humans, Models, Theoretical, Ownership, Prognosis, Software, Mobile Applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review