PreprintCited by 3,070COVID-19 Clinical Research StudiesStatistical Methods in EpidemiologyCOVID-19 diagnosis using AI

Systematic review and critical appraisal of prediction models for diagnosis and prognosis of COVID-19 infection

Open access
Wynants, Laure, Van Calster, Ben, Bonten, Marc MJ, Collins, Gary S, Debray, Thomas PA, De Vos, Maarten, Haller, Maria C., Heinze, Georg +8 more·2020·BMJ

Preprint — published as 10.1136/bmj.m1328 · Paper page

Abstract

Objective: To review and critically appraise published and preprint reports of models that aim to predict either (i) presence of existing COVID-19 infection, (ii) future complications in individuals already diagnosed with COVID-19, or (iii) models to identify individuals at high risk for COVID-19 in the general population. Design: Rapid systematic review and critical appraisal of prediction models for diagnosis or prognosis of COVID-19 infection. Data sources: PubMed, EMBASE via Ovid, Arxiv, medRxiv and bioRxiv until 24 th March 2020. Study selection: Studies that developed or validated a multivariable COVID-19 related prediction model. Two authors independently screened titles, abstracts and full text. Data extraction: Data from included studies were extracted independently by at least two authors based on the CHARMS checklist, and risk of bias was assessed using PROBAST. Data were extracted on various domains including the participants, predictors, outcomes, data analysis, and prediction model performance. Results: 2696 titles were screened. Of these, 27 studies describing 31 prediction models were included for data extraction and critical appraisal. We identified three models to predict hospital admission from pneumonia and other events (as a proxy for covid-19 pneumonia) in the general population; 18 diagnostic models to detect COVID-19 infection in symptomatic individuals (13 of which were machine learning utilising computed tomography (CT) results); and ten prognostic models for predicting mortality risk, progression to a severe state, or length of hospital stay. Only one of these studies used data on COVID-19 cases outside of China. Most reported predictors of presence of COVID-19 in suspected patients included age, body temperature, and signs and symptoms. Most reported predictors of severe prognosis in infected patients included age, sex, features derived from CT, C-reactive protein, lactic dehydrogenase, and lymphocyte count. Estimated C-index estimates for the prediction models ranged from 0.73 to 0.81 in those for the general population (reported for all 3 general population models), from 0.81 to > 0.99 in those for diagnosis (reported for 13 of the 18 diagnostic models), and from 0.85 to 0.98 in those for prognosis (reported for 6 of the 10 prognostic models). All studies were rated at high risk of bias, mostly because of non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, and poor statistical analysis, including high risk of model overfitting. Reporting quality varied substantially between studies. A description of the study population and intended use of the models was absent in almost all reports, and calibration of predictions was rarely assessed. Conclusion: COVID-19 related prediction models are quickly entering the academic literature, to support medical decision making at a time where this is urgently needed. Our review indicates proposed models are poorly reported and at high risk of bias. Thus, their reported performance is likely optimistic and using them to support medical decision making is not advised. We call for immediate sharing of the individual participant data from COVID-19 studies to support collaborative efforts in building more rigorously developed prediction models and validating (evaluating) existing models. The aforementioned predictors identified in multiple included studies could be considered as candidate predictors for new models. We also stress the need to follow methodological guidance when developing and validating prediction models, as unreliable predictions may cause more harm than benefit when used to guide clinical decisions. Finally, studies should adhere to the TRIPOD statement to facilitate validating, appraising, advocating and clinically using the reported models. Systematic review registration protocol: osf.io/ehc47/ , registration: osf.io/wy245 Summary boxes: What is already known on this topic: - The sharp recent increase in COVID-19 infections has put a strain on healthcare systems worldwide, necessitating efficient early detection, diagnosis of patients suspected of the infection and prognostication of COVID-19 confirmed cases. - Viral nucleic acid testing and chest CT are standard methods for diagnosing COVID-19, but are time-consuming. - Earlier reports suggest that the elderly, patients with comorbidity (COPD, cardiovascular disease, hypertension), and patients presenting with dyspnoea are vulnerable to more severe morbidity and mortality after COVID-19 infection. What this study adds: - We identified three models to predict hospital admission from pneumonia and other events (as a proxy for COVID-19 pneumonia) in the general population. - We identified 18 diagnostic models for COVID-19 detection in symptomatic patients. - 13 of these were machine learning models based on CT images. - We identified ten prognostic models for COVID-19 infected patients, of which six aimed to predict mortality risk in confirmed or suspected COVID-19 patients, two aimed to predict progression to a severe or critical state, and two aimed to predict a hospital stay of more than 10 days from admission. - Included studies were poorly reported compromising their subsequent appraisal, and recommendation for use in daily practice. All studies were appraised at high risk of bias, raising concern that the models may be flawed and perform poorly when applied in practice, such that their predictions may be unreliable.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2020.03.24.20041020

Cites

472 of 472 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The quality of the research is not universally adequate for summarization into policy.

    “Medical guidelines rely on comprehensive systematic reviews, but the medical literature is growing at breakneck pace and the quality of the research is not universally adequate for summarization into policy 38 .”

    From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [38]

    The Abstract Conclusion says the reviewed COVID-19 prediction models were poorly reported and at high risk of bias, and should not be used in current practice. The Study limitations section also notes that new models were rapidly entering the literature. This supports a concern about research quality and pace in that specific field, but the paper does not establish the broader claim about medical research generally being inadequate for summarization into policy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured
  • The medical literature is growing at breakneck pace.

    “Medical guidelines rely on comprehensive systematic reviews, but the medical literature is growing at breakneck pace and the quality of the research is not universally adequate for summarization into policy 38 .”

    From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [38]

    The Discussion’s Study limitations section says new publications on COVID-19 prediction models were rapidly entering the medical literature, making the review unable to remain fully up to date. This supports rapid growth in that specific field, but not the broader claim that medical literature as a whole is growing at that pace. (LLM, conf=0.84)

    Experimental — accuracy not yet measured
  • Within the first 15 months of the covid-19 pandemic, 381 prognostic prediction models for the disease were published.

    “within the first 15 months of the covid-19 pandemic, 381 prognostic prediction models for the disease were published.”

    From PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods (2025) · cites this paper as [5]

    The Results section reports 16 COVID-19 prognostic models identified in a search updated through 7 April 2020. It supports the existence of published prognostic models, but not the claim that 381 were published within the pandemic’s first 15 months; that later period is outside the review’s coverage. (LLM, conf=0.86)

    Experimental — accuracy not yet measured
  • Medical guidelines rely on comprehensive systematic reviews.

    “Medical guidelines rely on comprehensive systematic reviews, but the medical literature is growing at breakneck pace and the quality of the research is not universally adequate for summarization into policy 38 .”

    From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [38]

    The Methods describe a systematic review, and the Discussion (‘Implications for practice’; page unavailable) says it aims to provide information for healthcare decision makers and professionals. Neither passage addresses whether medical guidelines rely on comprehensive systematic reviews; a search for guidelines and policy found no passage establishing that relationship. (LLM, conf=0.72)

    Experimental — accuracy not yet measured
  • At least 731 diagnostic and prognostic prediction model studies on covid-19 were published during the first 12 months of the pandemic.

    “At least 731 diagnostic and prognostic prediction model studies on covid-19 were published during the first 12 months of the pandemic.9”

    From TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods (2024) · cites this paper as [9]

    The Results (page not provided) report 51 studies describing 66 models from searches through 7 April 2020. Our copy does not address how many prediction-model studies were published during the pandemic’s first 12 months, so it cannot verify the figure of at least 731; the earlier count does not contradict it. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

1 retracted

Flagged first, then by citations

Research works printed but not linked: 1

  1. [57]Tang Z, Zhao W, Xie X, et al. Severity assessment of coronavirus disease 2019 (covid-19) using quantitative features from chest CT images. arXiv e-prints [Preprint] 2020. https://ui.adsabs.harvard.edu/abs/2020arXiv200311988T.

Cited by

Sorted by citations

Authors
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., Schuit, E., Smits, L. J., Snell, K. I., Steyerberg, E. W., Wallisch, C., & van Smeden, M.
Affiliations
  • Maastricht University (NL)
  • Utrecht University (NL)
  • Krankenhaus der Elisabethinen (AT)
  • Medical University of Vienna (AT)
  • Keele University (GB)
  • Leiden University Medical Center (NL)
  • Humboldt-Universität zu Berlin (DE)
Corresponding author
Wynants, Laure
Venue
BMJ
Publisher
openRxiv
Published
2020-03-27
Volume
369
Pages
m1328-m1328
ISSN
1756-1833 (ISSN-L)
OpenAlex
W3014524604
Cited by
3,070
References
433
Citations by year
2026: 191 · 2025: 185 · 2024: 261 · 2023: 402 · 2022: 619 · 2021: 880 · 2020: 531
FWCI
28.74
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • BMJ (publishedVersion, cc-by)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Amsterdam UMC (VU Amsterdam) - Institutional Repository (publishedVersion, other-oa)
  • University of Birmingham Research Portal (University of Birmingham) (publishedVersion)
  • St George's Online Research Archive (St George's University of London) (acceptedVersion)
  • Leiden Repository (Leiden University) (submittedVersion, cc-by)
  • Aberdeen University Research Archive (Aberdeen University) (submittedVersion, cc-by)
  • Keele Research Repository (Keele University) (submittedVersion, cc-by)
  • Lirias (KU Leuven) (submittedVersion, cc-by)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
  • Research Publications (Maastricht University) (submittedVersion, cc-by)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • HighWire Press Open Archive (submittedVersion)
  • Pure Amsterdam UMC (submittedVersion, cc-by)
  • Research Explorer (The University of Manchester) (submittedVersion, cc-by)
  • Research Publications (Maastricht University) (submittedVersion, other-oa)
Funders
  • Cancer Research UK (27294, 16895, C49297/A27294, C49297)
  • National Institute for Health and Care Research (SPCR-142, COVID-19, C49297/A27294)
  • Department of Health and Social Care (COVID-19)
  • European Commission (825746, HORIZON2020, COVID-19)
  • ZonMw (91617050, 10430012010001, grant 91617050)
  • Fonds Wetenschappelijk Onderzoek (G0B4716N)
  • KU Leuven (C24/15/037)
  • Vlaamse regering (C24/15/037)
  • Oxford Health NHS Foundation Trust
  • National Institutes of Health (COVID-19)
  • NIHR Oxford Biomedical Research Centre (C49297/ A27294)
  • NIHR School for Primary Care Research
  • National Heart, Lung, and Blood Institute (R00HL141678)
  • National Institute of General Medical Sciences (P20GM125498)
  • Interreg
Topics
  • Health Sciences › Medicine › Infectious Diseases › COVID-19 Clinical Research Studies
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
Keywords
COVID-19 pandemic, prediction model, diagnostic model, prognostic model, risk prediction, model validation, risk of bias, concordance index, diagnostic imaging, disease severity, mortality prediction, intensive care unit admission, hospitalization risk, overfitting, reporting quality, preprints, charm, PROBAST
MeSH terms
COVID-19, Hospitalization, Humans, Models, Theoretical, Pneumonia, Viral, Prognosis, Multivariate Analysis, Coronavirus, Coronavirus Infections, Disease Progression, Pandemics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review