Prognostic models for COVID-19 needed updating to warrant transportability over time and space
Open accessAbstract
Background: Supporting decisions for patients who present to the emergency department (ED) with COVID-19 requires accurate prognostication. We aimed to evaluate prognostic models for predicting outcomes in hospitalized patients with COVID-19, in different locations and across time. Methods: We included patients who presented to the ED with suspected COVID-19 and were admitted to 12 hospitals in the New York City (NYC) area and 4 large Dutch hospitals. We used second-wave patients who presented between September and December 2020 (2137 and 3252 in NYC and the Netherlands, respectively) to evaluate models that were developed on first-wave patients who presented between March and August 2020 (12,163 and 5831). We evaluated two prognostic models for in-hospital death: The Northwell COVID-19 Survival (NOCOS) model was developed on NYC data and the COVID Outcome Prediction in the Emergency Department (COPE) model was developed on Dutch data. These models were validated on subsequent second-wave data at the same site (temporal validation) and at the other site (geographic validation). We assessed model performance by the Area Under the receiver operating characteristic Curve (AUC), by the E-statistic, and by net benefit. Results: Twenty-eight-day mortality was considerably higher in the NYC first-wave data (21.0%), compared to the second-wave (10.1%) and the Dutch data (first wave 10.8%; second wave 10.0%). COPE discriminated well at temporal validation (AUC 0.82), with excellent calibration (E-statistic 0.8%). At geographic validation, discrimination was satisfactory (AUC 0.78), but with moderate over-prediction of mortality risk, particularly in higher-risk patients (E-statistic 2.9%). While discrimination was adequate when NOCOS was tested on second-wave NYC data (AUC 0.77), NOCOS systematically overestimated the mortality risk (E-statistic 5.1%). Discrimination in the Dutch data was good (AUC 0.81), but with over-prediction of risk, particularly in lower-risk patients (E-statistic 4.0%). Recalibration of COPE and NOCOS led to limited net benefit improvement in Dutch data, but to substantial net benefit improvement in NYC data. Conclusions: NOCOS performed moderately worse than COPE, probably reflecting unique aspects of the early pandemic in NYC. Frequent updating of prognostic models is likely to be required for transportability over time and space during a dynamic pandemic.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CDS tool performance has been shown to drift when validated in hospitals and regions outside of the development sites and over time.
“Importantly, CDS tool performance has been shown to drift when validated in hospitals and regions outside of the development sites 14,15 and over time.16,17”
From A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools (2026) · cites this paper as [14]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
Zhou, F., Yu, T., Du, R., Fan, G., Liu, Y., Liu, Z., Xiang, J., Wang, Y., Song, B., Gu, X., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(20)30566-3 · Cited by 27,263 · see citing works
<b>mice</b>: Multivariate Imputation by Chained Equations in<i>R</i>
Buuren, S. V., & Groothuis-Oudshoorn, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i03 · Cited by 13,870 · see citing works
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270 · see citing works
Richardson, S., Hirsch, J. S., Narasimhan, M., Crawford, J. M., McGinn, T., Davidson, K. W., and the Northwell COVID-19 Research Consortium, Barnaby, D. P., Becker, L. B., Chelico, J. D., et al. (2020) · JAMA · doi:10.1001/jama.2020.6775 · Cited by 9,254 · see citing works
Self-citationDecision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · see citing works
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group† (2019) · Annals of Internal Medicine · doi:10.7326/m18-1376 · Cited by 2,984 · see citing works
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · see citing works
Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669 · see citing works
Prediction models need appropriate internal, internal-external, and external validation
Steyerberg, E. W., & Harrell, F. E. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.005 · Cited by 1,304 · see citing works
Covid-19 Vaccines — Immunity, Variants, Boosters
Barouch, D. H. (2022) · New England Journal of Medicine · doi:10.1056/nejmra2206573 · Cited by 434 · see citing works
Forecasting for COVID-19 has failed
Ioannidis, J. P. A., Cripps, S., & Tanner, M. A. (2020) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2020.08.004 · Cited by 432 · see citing works
Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371 · see citing works
Big Data and Predictive Analytics
Shah, N. D., Steyerberg, E. W., & Kent, D. M. (2018) · JAMA · doi:10.1001/jama.2018.5602 · Cited by 244 · see citing works
Self-citationDetection of calibration drift in clinical prediction models to inform model updating
Davis, S. E., Greevy, R. A., Lasko, T. A., Walsh, C. G., & Matheny, M. E. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103611 · Cited by 160 · see citing works
A review of statistical updating methods for clinical prediction models
Su, T.-L., Jaki, T., Hickey, G. L., Buchan, I., & Sperrin, M. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280215626466 · Cited by 147 · see citing works
Jenkins, D. A., Martin, G. P., Sperrin, M., Riley, R. D., Debray, T. P. A., Collins, G. S., & Peek, N. (2021) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00090-3 · Cited by 131 · see citing works
Wessler, B. S., Nelson, J., Park, J. G., McGinnes, H., Gulati, G., Brazil, R., Van Calster, B., van Klaveren, D., Venema, E., Steyerberg, E., et al. (2021) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.007858 · Cited by 83 · see citing works
Self-citationVenema, E., Wessler, B. S., Paulus, J. K., Salah, R., Raman, G., Leung, L. Y., Koethe, B. C., Nelson, J., Park, J. G., Klaveren, D. V., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.017 · Cited by 65 · see citing works
Self-citationLombardi, Y., Azoyan, L., Szychowiak, P., Bellamine, A., Lemaître, G., Bernaux, M., Daniel, C., Leblanc, J., Riller, Q., Steichen, O., et al. (2021) · Intensive Care Medicine · doi:10.1007/s00134-021-06524-w · Cited by 61 · see citing works
Gulati, G., Upshaw, J., Wessler, B. S., Brazil, R. J., Nelson, J., Klaveren, D. V., Lundquist, C. M., Park, J. G., McGinnes, H., Steyerberg, E. W., et al. (2022) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.008487 · Cited by 60 · see citing works
Self-citationDynamic Prediction Modeling Approaches for Cardiac Surgery
Hickey, G., Grant, S., Caiado, C., Kendall, S., Dunning, J., Poullis, M., Buchan, I. E., & Bridgewater, B. (2013) · Circulation. Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.111.000012 · Cited by 55 · see citing works
A predictive model to estimate survival of hospitalized COVID-19 patients from admission dataPreprint
Levy, T. J., Richardson, S. I., Coppa, K., Barnaby, D. P., McGinn, T. G., Becker, L. B., Davidson, K. W., Hirsch, J. S., Zanos, T. P., Cohen, S. L., et al. (2020) · medRxiv · doi:10.1101/2020.04.22.20075416 · Cited by 52 · see citing works
Self-citationA new concordance measure for risk prediction models in external validation settings
van Klaveren, D., Gönen, M., Steyerberg, E. W., & Vergouwe, Y. (2016) · Statistics in Medicine · doi:10.1002/sim.6997 · Cited by 48 · see citing works
Chi, S., Tian, Y., Wang, F., Zhou, T., Jin, S., & Li, J. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102256 · Cited by 46 · see citing works
Clarification of Mortality Rate and Data in Abstract, Results, and Table 2
(2020) · JAMA · doi:10.1001/jama.2020.7681 · Cited by 37 · see citing works
Improved Prediction by Dynamic Modeling
Siregar, S., Nieboer, D., Vergouwe, Y., Versteegh, M. I. M., Noyez, L., Vonk, A. B. A., Steyerberg, E. W., & Takkenberg, J. J. M. (2016) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.114.001645 · Cited by 26 · see citing works
The Global Response to the COVID-19 Pandemic
Nkengasong, J., Iwasaki, A., Victora, C. G., Oh, J., Gao, G. F., Agrawal, A., Drosten, C., Söderberg‐Nauclér, C., López‐Collazo, E., Pollock, A. M., et al. (2020) · Med · doi:10.1016/j.medj.2020.12.003 · Cited by 20 · see citing works
Klaveren, D. V., Rekkas, A., Alsma, J., Verdonschot, R. J. C. G., Koning, D., Kamps, M. J. A., Dormans, T. P. J., Stassen, R. C., Weijer, S., Arnold, K.-S., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-051468 · Cited by 18 · see citing works
Self-citationComparison of dynamic updating strategies for clinical prediction models
Schnellinger, E. M., Yang, W. P., & Kimmel, S. E. (2021) · Diagnostic and Prognostic Research · doi:10.1186/s41512-021-00110-w · Cited by 18 · see citing works
Helmrich, I. R., Mikolić, A., Kent, D. M., Lingsma, H. F., Wynants, L., Steyerberg, E. W., & Klaveren, D. V. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00122-0 · Cited by 17 · see citing works
Self-citationCazeau, N. (2020) · Clinical journal of oncology nursing · doi:10.1188/20.cjon.472-474 · Cited by 8 · see citing works
Flagged first, then by citations
1 of the 34 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [28]Gulati G, Upshaw JN, Wessler BS, Brazil RJ, Nelson J, van Klaveren D, et al. The Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Large-Scale Independent External Validations of 104 Unique Models. Circ Cardiovasc Qual Outcomes. 2022;15(4):e008487. PMC9015037 35354282
Cited by
Sorted by citations
Basile, M., Helmrich, I., Park, J. G., Polo, J., Rietjens, J., van Klaveren, D., Zanos, T. P., Nelson, J., Lingsma, H. F., Kent, D., et al. (2023) · Medical decision making · doi:10.1177/0272989x231152852 · Cited by 5 · see citing works
Self-citationA Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools
Patel, H., Crusco, S., Hansen, D., Morales, J., Patale, T., Chiuzan, C., Sands, M., Naidich, J., Myers, K., Davidson, K. W., et al. (2026) · Journal of General Internal Medicine · doi:10.1007/s11606-026-10381-y · Cited by 4 · see citing works
Self-citationDamen, J. A., Arshi, B., van Smeden, M., Bertagnolio, S., Diaz, J. V., Silva, R., Thwin, S. S., Wynants, L., & Moons, K. G. M. (2024) · Diagnostic and Prognostic Research · doi:10.1186/s41512-024-00181-5 · Cited by 4 · see citing works
Research Trends in Proteomic Studies Using Serum from COVID-19 Patients: A Bibliometric Analysis
Amorim, É. A. D. F., Neto, R. N. M., Souza, A. V., Martinez, C. G., Zagmignan, A., & da Silva, L. C. N. (2024) · Current Medicinal Chemistry · doi:10.2174/0109298673286915240329063441 · Cited by 1 · see citing works
Williams, R. J., Brintz, B. J., Grisel, N., Otterloo, J. V., Kieber-Emmons, A. M., Ebell, M. H., Leung, D. T., & Lopansri, B. K. (2026) · Open Forum Infectious Diseases · doi:10.1093/ofid/ofag226
- Authors
- van Klaveren, D., Zanos, T. P., Nelson, J., Levy, T. J., Park, J. G., Retel Helmrich, I. R. A., Rietjens, J. A. C., Basile, M. J., Hajizadeh, N., Lingsma, H. F., & Kent, D. M.
- Affiliations
- Northwell Health (US)
- Tufts Medical Center (US)
- Erasmus MC (NL)
- ORCID
- Venue
- BMC Medicine
- Published
- 2022-11-23
- Volume
- 20
- Issue
- 1
- Pages
- 456-456
- ISSN
- 1741-7015 (ISSN-L)
- PMC
- PMC9686462
- OpenAlex
- W4309890846
- Cited by
- 5
- References
- 34
- Citations by year
- 2026: 2 · 2024: 2 · 2023: 1
- FWCI
- 0.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, cc-by)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Patient-Centered Outcomes Research Institute (ME-1606-35555)
- ZonMw (10430 01 201 0019)
- Topics
- Health Sciences › Medicine › Infectious Diseases › COVID-19 Clinical Research Studies
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
- Keywords
- COVID-19 pandemic, prognostic model, in-hospital mortality prediction, model portability, temporal validation, model recalibration, second wave of COVID-19, emergency department, model updating, New York City, Netherlands, net benefit, discrimination and calibration, hospitalized patients
- MeSH terms
- COVID-19, Humans, New York City, Prognosis, ROC Curve, Hospital Mortality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
