Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients
Open accessAbstract
Clinical prognostic models can assist patient care decisions. However, their performance can drift over time and location, necessitating model monitoring and updating. Despite rapid and significant changes during the pandemic, prognostic models for COVID-19 patients do not currently account for these drifts. We develop a framework for continuously monitoring and updating prognostic models and apply it to predict 28-day survival in COVID-19 patients. We use demographic, laboratory, and clinical data from electronic health records of 34912 hospitalized COVID-19 patients from March 2020 until May 2022 and compare three modeling methods. Model calibration performance drift is immediately detected with minor fluctuations in discrimination. The overall calibration on the prospective validation cohort is significantly improved when comparing the dynamically updated models against their static counterparts. Our findings suggest that, using this framework, models remain accurate and well-calibrated across various waves, variants, race and sex and yield positive net-benefits.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CDS tool performance has been shown to drift when validated in hospitals and regions outside of the development sites and over time.
“Importantly, CDS tool performance has been shown to drift when validated in hospitals and regions outside of the development sites 14,15 and over time.16,17”
From A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools (2026) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1× · see citing works
“…shrinkage and selection operator (LASSO) penalization (the Northwell COVID-19 Survival Calculator (NOCOS)) 42 , logistic regression (LR) with LASSO penalization 43 , and extreme gradient boosted decision tree (XGBoost) 44 — were trained on patients from the development cohort for 28-day survival.”
MethodThe meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
<b>mice</b>: Multivariate Imputation by Chained Equations in<i>R</i>
Buuren, S. V., & Groothuis-Oudshoorn, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i03 · Cited by 13,870 · see citing works
A study of the behavior of several methods for balancing machine learning training data
Batista, G. E. A. P. A., Prati, R., & Monard, M. (2004) · SKDD · doi:10.1145/1007730.1007735 · Cited by 4,271 · cited 1× · see citing works
“The fi ve imputed development cohorts were combined via concatenation for model development 63 and weighted so as not to arti fi cially increase the sample size, and the minority class (patients that died) was randomly oversampled with replacement 64,65 to correct for class imbalance.”
MethodEuropean system for cardiac operative risk evaluation (EuroSCORE)
Nashef, S. A. M., Roques, F., Michel, P., Gauducheau, E., Lemeshow, S., & Salamon, R. (1999) · European Journal of Cardio-Thoracic Surgery · doi:10.1016/s1010-7940(99)00134-7 · Cited by 3,362 · cited 1× · see citing works
“For example, EuroSCORE was initially developed to predict mortality after heartsurgery 20 , and it wasupdated twelve years later 14 .”
MethodWynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · cited 1× · see citing works
“Since the ongoing novel coronavirus pandemic is a situation in which many prognostic models were developed 19,36,37 on geographically distinct populations alongside rapid changes in patient populations and standards of care 38 , it is unsurprising that a growing number of studies are reporting…”
BackgroundLim, W. S. (2003) · Thorax · doi:10.1136/thorax.58.5.377 · Cited by 3,066 · see citing works
Nashef, S. A. M., Roques, F., Sharples, L. D., Nilsson, J., Smith, C., Goldstone, A. R., & Lockowandt, U. (2012) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezs043 · Cited by 2,971 · cited 1× · see citing works
“For example, EuroSCORE was initially developed to predict mortality after heartsurgery 20 , and it wasupdated twelve years later 14 .”
MethodCalculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · cited 1× · see citing works
“…balance for this tradeoff, our approach was to determine the size of the window based on a combination of formulas that estimate the proper sample size for LR models 52 and to perform a hyperparameter optimization that estimates the proper sample size for the custom GLM model (Supplementary Fig.”
MethodCalibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · see citing works
Hippisley-Cox, J., Coupland, C., & Brindle, P. (2017) · BMJ · doi:10.1136/bmj.j2099 · Cited by 1,703 · see citing works
Clinical Prediction Models—a Practical Approach to Development, Validation and Updating
Bedogni, G. (2009) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2009.00614_16.x · Cited by 1,526 · cited 5× · see citing works
“Outcomes were also included in the imputation models as recommended by Steyerberg in 7.4.3 of Clinical Prediction Models Second Edition since missing values were imputed using random draws 13 .”
MethodInfluentialPrognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · see citing works
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · see citing works
Menze, B. H., Kelm, B. M., Masuch, R., Himmelreich, U., Bachert, P., Petrich, W. H., & Hamprecht, F. A. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-213 · Cited by 1,315 · cited 1× · see citing works
“Six predictors were selected for the XGBoost models by including all candidate predictors, ranking the predictors by their importance that is based on averaging changes in the node risk due to splits on every predictor 66 , and then retraining the models using only the six most important predictors.”
MethodA simple, step-by-step guide to interpreting decision curve analysis
Vickers, A. J., van Calster, B., & Steyerberg, E. W. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0064-7 · Cited by 1,174 · see citing works
Hippisley-Cox, J., Coupland, C., Vinogradova, Y., Robson, J., May, M., & Brindle, P. (2007) · British medical journal · doi:10.1136/bmj.39261.471806.55 · Cited by 1,070 · cited 1× · see citing works
“Another clinical prognostic model, QRISK, which predicts the risk of a patient developing a heart attack or stroke within ten years is updated annually 15,21,22 .”
Factors associated with death in critically ill patients with coronavirus disease 2019 in the US
Gupta, S., Hayek, S. S., Wang, W., Chan, L., Mathews, K. S., Melamed, M. L., Brenner, S. K., Leonberg-Yoo, A., Schenck, E. J., Radbel, J., et al. (2020) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2020.3596 · Cited by 957 · see citing works
Viral and host factors related to the clinical outcome of COVID-19
Zhang, X., Tan, Y., Ling, Y., Lü, G., Liu, F., Yi, Z., Jia, X., Wu, M., Shi, B., Xu, S., et al. (2020) · Nature · doi:10.1038/s41586-020-2355-0 · Cited by 860 · see citing works
Jones, A. E., Trzeciak, S., & Kline, J. A. (2009) · Critical Care Medicine · doi:10.1097/ccm.0b013e31819def97 · Cited by 853 · see citing works
A prognostic index in primary breast cancer.
Haybittle, J., Blamey, R., Elston, C., Johnson, J., Doyle, P., Campbell, F., Nicholson, R. I., & Griffiths, K. (1982) · British Journal of Cancer · doi:10.1038/bjc.1982.62 · Cited by 758 · see citing works
Area under the Precision-Recall Curve: Point Estimates and Confidence Intervals
Boyd, K., Eng, K., & Page, D. (2013) · ECML/PKDD · doi:10.1007/978-3-642-40994-3_29 · Cited by 549 · see citing works
Ayyadevara, V. K. (2018) · Apress eBooks · doi:10.1007/978-1-4842-3564-5_6 · Cited by 479 · see citing works
Nonlinear Dynamic Modeling of Physiological Systems
Marmarelis, V. (2004) · Wiley eBooks · doi:10.1002/9780471679370 · Cited by 416 · cited 1× · see citing works
“Thereisaninherenttradeoffbetween stationarity and sample size in the update cohort 51 .”
BackgroundIftimie, S. M., López-Azcona, A. F., Vallverdú, I., Hernández-Flix, S., de Febrer, G., Parra, S., Hernández‐Aguilera, A., Riu, F., Joven, J., Andreychuk, N., et al. (2021) · PLoS ONE · doi:10.1371/journal.pone.0248029 · Cited by 343 · see citing works
The area under the precision‐recall curve as a performance metric for rare binary events
Sofaer, H., Hoeting, J. A., & Jarnevich, C. (2019) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13140 · Cited by 322 · cited 1× · see citing works
“AUPR is a measure that is well-suited for imbalanced data, and its values range from the sample prevalence (indicating random performance) to 1 (indicating perfect classi fi cation) 67 .”
BackgroundAustin, P. C., & Steyerberg, E. W. (2019) · Statistics in Medicine · doi:10.1002/sim.8281 · Cited by 311 · cited 1× · see citing works
“ICI is the expected error between the actual and ideal calibration curves and approaches 0 when there is perfect calibration 68 .”
BackgroundCovid-19 Vaccine Effectiveness in New York State
Rosenberg, E., Dorabawila, V., Easton, D., Bauer, U. E., Kumar, J., Hoen, R., Hoefer, D., Wu, M., Lutterloh, E. C., Conroy, M., et al. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2116063 · Cited by 262 · see citing works
Collins, P. G. S., & Altman, D. G. (2012) · BMJ · doi:10.1136/bmj.e4181 · Cited by 245 · see citing works
Trends in COVID-19 risk-adjusted mortality rates
Horwitz, L. I., Jones, S. A., Cerfolio, R. J., Francois, F., Greco, J., Rudy, B., & Petrilli, C. M. (2021) · Journal of Hospital Medicine · doi:10.12788/jhm.3552 · Cited by 229 · see citing works
Dennis, J. M., McGovern, A. P., Vollmer, S. J., & Mateen, B. A. (2021) · Critical Care Medicine · doi:10.1097/ccm.0000000000004747 · Cited by 214 · see citing works
Toward an Information Infrastructure for Global Health Improvement
Friedman, C. P., Rubin, J. C., & Sullivan, K. (2017) · Yearbook of Medical Informatics · doi:10.15265/iy-2017-004 · Cited by 214 · see citing works
Gupta, R. K., Marks, M., Samuels, T. H. A., Luintel, A., Rampling, T., Chowdhury, H., Quartagno, M., Nair, A., Lipman, M. C. I., Abubakar, I. I., et al. (2020) · European Respiratory Journal · doi:10.1183/13993003.03498-2020 · Cited by 189 · see citing works
Detection of calibration drift in clinical prediction models to inform model updating
Davis, S. E., Greevy, R. A., Lasko, T. A., Walsh, C. G., & Matheny, M. E. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103611 · Cited by 160 · cited 2× · see citing works
“Other methods in the literature for correcting model performance calibration drift include a closed test procedure 54 , applying bootstrap resampling while evaluating the updating methods and scoring rules 10 , and dynamically adjusting the window lengths 12 .”
MethodA closed testing procedure to select an appropriate method for updating prediction models
Vergouwe, Y., Nieboer, D., Oostenbrink, R., Debray, T. P. A., Murray, G. D., Kattan, M. W., Koffijberg, H., Moons, K. G. M., & Steyerberg, E. W. (2016) · Statistics in Medicine · doi:10.1002/sim.7179 · Cited by 159 · cited 2× · see citing works
“Other methods in the literature for correcting model performance calibration drift include a closed test procedure 54 , applying bootstrap resampling while evaluating the updating methods and scoring rules 10 , and dynamically adjusting the window lengths 12 .”
BackgroundMethodCase Studies in Emergency Medicine
Ryan, J. (1997) · doi:10.1136/emj.14.3.191-c · Cited by 149 · cited 2×
“6).”
MethodA review of statistical updating methods for clinical prediction models
Su, T.-L., Jaki, T., Hickey, G. L., Buchan, I., & Sperrin, M. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280215626466 · Cited by 147 · see citing works
Jenkins, D. A., Martin, G. P., Sperrin, M., Riley, R. D., Debray, T. P. A., Collins, G. S., & Peek, N. (2021) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00090-3 · Cited by 131 · cited 1× · see citing works
“Dynamic models with continuous model surveillance have the bene fi t of increased responsiveness but also add complexity 24,49 to the validation steps since there is no standardized validation methodology 7 – 10 .”
MethodJarman, B., Pieter, D., van der Veen, A., Kool, R. B., Aylin, P., Bottle, A., Westert, G., & Jones, S. (2010) · BMJ Quality & Safety · doi:10.1136/qshc.2009.032953 · Cited by 129 · see citing works
A nonparametric updating method to correct clinical prediction model drift
Davis, S. E., Greevy, R., Fonnesbeck, C., Lasko, T., Walsh, C. G., & Matheny, M. E. (2019) · J. Am. Medical Informatics Assoc. · doi:10.1093/jamia/ocz127 · Cited by 112 · cited 3× · see citing works
“Dynamic models with continuous model surveillance have the bene fi t of increased responsiveness but also add complexity 24,49 to the validation steps since there is no standardized validation methodology 7 – 10 .”
MethodBrajer, N., Cozzi, B., Gao, M., Nichols, M., Revoir, M., Balu, S., Futoma, J., Bae, J., Setji, N. P., Hernandez, A. F., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.20733 · Cited by 107 · see citing works
Current and evolving standards of care for patients with ARDS
Menk, M., Estenssoro, E., Sahetya, S. K., Neto, A. S., Sinha, P., Slutsky, A. S., Summers, C., Yoshida, T., Bein, T., & Ferguson, N. D. (2020) · Intensive Care Medicine · doi:10.1007/s00134-020-06299-6 · Cited by 104 · cited 1× · see citing works
“…models were developed 19,36,37 on geographically distinct populations alongside rapid changes in patient populations and standards of care 38 , it is unsurprising that a growing number of studies are reporting drifts in the performance of such models across different geographic regions…”
BackgroundDynamic models to predict health outcomes: current status and methodological challenges
Jenkins, D. A., Sperrin, M., Martin, G., & Peek, N. (2018) · Diagnostic and Prognostic Research · doi:10.1186/s41512-018-0045-2 · Cited by 96 · see citing works
Evolving changes in mortality of 13,301 critically ill adult patients with COVID-19 over 8 months
Kurtz, P., Bastos, L., Dantas, L. F., Zampieri, F., Soares, M., Hamacher, S., Salluh, J., & Bozza, F. (2021) · Intensive Care Medicine · doi:10.1007/s00134-021-06388-0 · Cited by 91 · cited 1× · see citing works
“A commonly reported change was related to the signi fi cant decrease of case mortality rate of hospitalized patients in certain areas 33 – 35 , as well as the number of intubated patients 35 and overall hospitalized patients between waves.”
BackgroundDynamic Logistic Regression and Dynamic Model Averaging for Binary Classification
McCormick, T. H., Raftery, A., Madigan, D., & Burd, R. S. (2011) · Biometrics · doi:10.1111/j.1541-0420.2011.01645.x · Cited by 88 · cited 4× · see citing works
“TheLRmodel also included a dynamic Bayesian logistic regression 23 updating method that was batch updated at each window position without a threshold.”
MethodInfluentialWessler, B. S., Nelson, J., Park, J. G., McGinnes, H., Gulati, G., Brazil, R., Van Calster, B., van Klaveren, D., Venema, E., Steyerberg, E., et al. (2021) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.007858 · Cited by 83 · see citing works
Handling missing predictor values when validating and applying a prediction model to new patients
Hoogland, J., van Barreveld, M., Debray, T. P. A., Reitsma, J. B., Verstraelen, T. E., Dijkgraaf, M. G. W., & Zwinderman, A. H. (2020) · Statistics in Medicine · doi:10.1002/sim.8682 · Cited by 82 · see citing works
Nikparvar, B., Rahman, M. M., Hatami, F., & Thill, J. (2021) · Scientific Reports · doi:10.1038/s41598-021-01119-3 · Cited by 73 · cited 1× · see citing works
“…geographically distinct populations alongside rapid changes in patient populations and standards of care 38 , it is unsurprising that a growing number of studies are reporting drifts in the performance of such models across different geographic regions 39 – 41 or different temporal windows 39,40 .”
BackgroundGulati, G., Upshaw, J., Wessler, B. S., Brazil, R. J., Nelson, J., Klaveren, D. V., Lundquist, C. M., Park, J. G., McGinnes, H., Steyerberg, E. W., et al. (2022) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.008487 · Cited by 60 · see citing works
Self-citationBarish, M., Bolourani, S., Lau, L. F., Shah, S., & Zanos, T. P. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00254-2 · Cited by 58 · cited 2× · see citing works
“External, and more speci fi cally geographic, validation of similar models has been limited and has demonstrated signi fi cant performance drifts 41,58 .”
BackgroundSelf-citationA validated, real-time prediction model for favorable outcomes in hospitalized COVID-19 patients
Razavian, N., Major, V. J., Sudarshan, M., Burk‐Rafel, J., Stella, P., Randhawa, H., Bilaloglu, S., Chen, J., Nguy, V., Wang, W., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00343-x · Cited by 57 · cited 1× · see citing works
“Since the ongoing novel coronavirus pandemic is a situation in which many prognostic models were developed 19,36,37 on geographically distinct populations alongside rapid changes in patient populations and standards of care 38 , it is unsurprising that a growing number of studies are reporting…”
BackgroundDynamic Prediction Modeling Approaches for Cardiac Surgery
Hickey, G., Grant, S., Caiado, C., Kendall, S., Dunning, J., Poullis, M., Buchan, I. E., & Bridgewater, B. (2013) · Circulation. Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.111.000012 · Cited by 55 · cited 4× · see citing works
“Dynamic models with continuous model surveillance have the bene fi t of increased responsiveness but also add complexity 24,49 to the validation steps since there is no standardized validation methodology 7 – 10 .”
MethodInfluentialVahedi, B., Karimzadeh, M., & Zoraghein, H. (2021) · Nature Communications · doi:10.1038/s41467-021-26742-6 · Cited by 49 · cited 1× · see citing works
“…geographically distinct populations alongside rapid changes in patient populations and standards of care 38 , it is unsurprising that a growing number of studies are reporting drifts in the performance of such models across different geographic regions 39 – 41 or different temporal windows 39,40 .”
BackgroundVartiainen, P., Jukarainen, S., Rhedin, S., Prinz, A., Hartonen, T., Vabalas, A., Viippola, E., Rodosthenous, R., Kuitunen, S., Liu, A., et al. (2023) · medRxiv · doi:10.1101/2023.02.23.23286237 · Cited by 41
Varotto, G., Susi, G., Tassi, L., Gozzo, F., Franceschetti, S., & Panzica, F. (2021) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2021.715421 · Cited by 39
Decreased in-hospital mortality in patients with COVID-19 pneumonia
Ciceri, F., Ruggeri, A., Lembo, R., Puglisi, R., Landoni, G., & Zangrillo, A. (2020) · Pathogens and Global Health · doi:10.1080/20477724.2020.1785782 · Cited by 39 · cited 1×
“We show that model updating is crucial, particularly in the setting of rapidly changing outcomes (e.g., survival), as in the course of the COVID-19 pandemic at our health system (Table 1) and elsewhere 47,48 .”
BackgroundRing, A. E., Battisti, N. M. L., Reed, M., Herbert, E., Morgan, J. L., Bradburn, M., Walters, S. J., Collins, K., Ward, S. E., Holmes, G. R., et al. (2021) · British Journal of Cancer · doi:10.1038/s41416-021-01388-9 · Cited by 36 · see citing works
Saban, M., Myers, V., & Wilf-Miron, R. (2021) · Preventive Medicine · doi:10.1016/j.ypmed.2021.106890 · Cited by 36 · see citing works
C-reactive protein levels associated with COVID-19 outcomes in the United States
Lentner, J., Adams, T., Knutson, V., Zeien, S. L., Abbas, H., Moosavi, R. S., Manuel, C., Wallace, T., Harmon, A., Waters, R. D., et al. (2021) · Journal of Osteopathic Medicine · doi:10.1515/jom-2021-0103 · Cited by 29 · cited 1×
“C-reactive protein serum was also included even though it was more than 50% missing because other studies found it to be predictive of mortality 59,60 .”
ResultRiley, R., Moons, K., Debray, T. P. A., Snell, K., Steyerberg, E., Altman, D., & Collins, G. (2019) · Prognosis Research in Health Care · doi:10.1093/med/9780198796619.003.0008 · Cited by 29
Austin, P., Lee, D. S., Ko, D. T., & White, I. (2019) · Circulation. Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.119.005927 · Cited by 23 · cited 1× · see citing works
“The fi ve imputed development cohorts were combined via concatenation for model development 63 and weighted so as not to arti fi cially increase the sample size, and the minority class (patients that died) was randomly oversampled with replacement 64,65 to correct for class imbalance.”
MethodVilloteau, A., Asfar, M., Otekpo, M., Loison, J., Gautier, J., & Annweiler, C. (2021) · PLoS ONE · doi:10.1371/journal.pone.0256931 · Cited by 22 · cited 1× · see citing works
“C-reactive protein serum was also included even though it was more than 50% missing because other studies found it to be predictive of mortality 59,60 .”
ResultTechnical aspects of clinical videoconferencing: a large scale review of the literature
Fatehi, F., Armfield, N. R., Dimitrijevic, M., & Gray, L. C. (2015) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x15571999 · Cited by 22 · see citing works
Comparison of dynamic updating strategies for clinical prediction models
Schnellinger, E. M., Yang, W. P., & Kimmel, S. E. (2021) · Diagnostic and Prognostic Research · doi:10.1186/s41512-021-00110-w · Cited by 18 · cited 5× · see citing works
“…a lag equaltothefollow-upperiodbeforeupdatingthemodeltoensurethat updating was done prospectively, unlike the approach by Schnellinger et al. that was inherently retrospective 11 , which we found to be overly optimistic (see hyperparameter optimization and causal model design in Supplementary Fig.”
BackgroundMethodResultInfluentialFeng, C., Kephart, G., & Juarez-colunga, E. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01441-4 · Cited by 17 · cited 1×
“This is likely because we constrained the models to only a few predictors or because the relationships in the data were mostly linear 55,56 .”
MethodCOVID Outcome Prediction in the Emergency Department (COPE)
van Klaveren, D., Rekkas, A., Alsma, J., Verdonschot, R., Koning, D., Kamps, M., Dormans, T., Stassen, R., Weijer, S., Arnold, K., et al. (2021) · medRxiv · doi:10.1101/2020.12.30.20249023 · Cited by 15 · cited 2×
“Clinical decisions should not be based only on the predicted probability of survival, but also be based on a risk-bene fi t analysis that may be dependent upon current hospital resources 37 .”
BackgroundSelf-citationFreeman, A., Watson, A., O'Regan, P., Wysocki, O., Burke, H., Freitas, A., Livingstone, R., Dushianthan, A., Celinski, M., Batchelor, J., et al. (2021) · Journal of Clinical Virology · doi:10.1016/j.jcv.2021.105031 · Cited by 14 · cited 1× · see citing works
“Moreover, virus variants, changing vaccination rates, waning of vaccine protection, and emergence of novel COVID-19 treatments also contributed to a highly dynamic environment for patient characteristics and outcomes 29 – 32 .”
BackgroundLazin, F. A. (2018) · Hrvatska i komparativna javna uprava · doi:10.31297/hkju.18.3.6 · Cited by 14
Trends in Covid-19 risk-adjusted mortality rates in a single health systemPreprint
Horwitz, L., Jones, S., Cerfolio, R., François, F., Greco, J., Rudy, B., & Petrilli, C. A. (2020) · medRxiv · doi:10.1101/2020.08.11.20172775 · Cited by 13 · see citing works
Performance of prediction models for COVID-19: the Caudine Forks of the external validation
Martin, G., Sperrin, M., & Sotgiu, G. (2020) · European Respiratory Journal · doi:10.1183/13993003.03728-2020 · Cited by 9 · see citing works
Yamada, G., Hayakawa, K., Asai, Y., Matsunaga, N., Ohtsu, H., Hojo, M., Hashimoto, M., Kobayashi, K., Sasaki, R., Okamoto, T., et al. (2021) · Journal of Infection and Chemotherapy · doi:10.1016/j.jiac.2021.04.008 · Cited by 6 · cited 1×
“While rapid and continuous geographic validation of a newly developed model is challenging 18,25 – 27 , temporal validation withinone site or institution can be feasible, due to continuous recording of patient data as part of the standard of care.”
BackgroundDynamic logistic state space prediction model for clinical decision making
Jiang, J., Yang, W. P., Schnellinger, E. M., Kimmel, S. E., & Guo, W. (2021) · Biometrics · doi:10.1111/biom.13593 · Cited by 4 · cited 1× · see citing works
“Inaddition,otherrecentmethodsthatsmoothlyupdate their parameters like the dynamiclogisticstate spacemodel 53 mayalso be valuable within a dynamic updating framework.”
Background
Flagged first, then by citations
8 of the 80 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- [6]Emergency Triage: Manchester Triage Group, 3rd Edition | Wiley. https://www.wiley.com/en-us/Emergency+Triage%3A+Manchester+Triage+Group%2C+3rd+Edition-p-9781118299067.
- [42]Levy, T. J. et al. Development and validation of a survival calculator for hospitalized patients with COVID-19. Preprint at https://www.medrxiv.org/content/10.1101/2020.04.22.20075416v3 (2020). · DOI
- [45]Pattern Recognition - 4th Edition. https://www.elsevier.com/books/pattern-recognition/koutroumbas/978-1-59749-272-0.
- [47]Horwitz, L. et al. Trends in Covid-19 risk-adjusted mortality rates. J Hosp Med.16, 90–92 (2021). 33147129
- [56]Yadaw AS Clinical features of COVID-19 mortality: development and validation of a clinical prediction model Lancet Digital Health 2020 2 e516 e525 32984797 PMC7508513
- [71]Vickers AJ Cronin AM Elkin EB Gonen M Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers BMC Med. Inf. Decis. Mak. 2008 8 53 PMC2611975 19036144
Cited by
Sorted by citations
Empirical data drift detection experiments on real-world medical imaging data
Kore, A., Abbasi Bavil, E., Subasri, V., Abdalla, M., Fine, B., Dolatabadi, E., & Abdalla, M. (2024) · Nature Communications · doi:10.1038/s41467-024-46142-w · Cited by 124 · see citing works
Monitoring performance of clinical artificial intelligence in health care: a scoping review
Andersen, E. S., Birk-Korch, J. B., Hansen, R. S., Fly, L. H., Röttger, R., Arcani, D. M. C., Brasen, C. L., Brandslund, I., & Madsen, J. S. (2024) · JBI Evidence Synthesis · doi:10.11124/jbies-24-00042 · Cited by 73 · see citing works
“38,47,57 Calibration Six sources described monitoring of calibration measures.”
Automatic correction of performance drift under acquisition shift in medical image classification
Roschewitz, M., Khara, G., Yearsley, J. E., Sharma, N., James, J. J., Ambrózay, É., Heroux, A. M., Kecskemethy, P., Rijken, T., & Glocker, B. (2023) · Nature Communications · doi:10.1038/s41467-023-42396-y · Cited by 56 · see citing works
Machine Learning Operations in Health Care: A Scoping Review
Rajagopal, A., Ayanian, S., Ryu, A. J., Qian, R., Legler, S. R., Peeler, E. A., Issa, M., Coons, T. J., & Kawamoto, K. (2024) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2024.06.009 · Cited by 52 · see citing works
“25,26 27-31 Davis et al 27 additionally developed a framework for selecting the least complex model update to ameliorate speci fi c types of drift detected in various simulations.”
González, R., Saha, A., Campbell, C. J. V., Nejat, P., Lokker, C., & Norgan, A. P. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100347 · Cited by 21 · see citing works
“133 If some major technical challenges are overcome, 134 – 137 and these steps can be done automatically, 122,132,138,139 a site-speci fi c autonomous endless self-learning process could eventually be developed.”
BackgroundBasile, M., Helmrich, I., Park, J. G., Polo, J., Rietjens, J., van Klaveren, D., Zanos, T. P., Nelson, J., Lingsma, H. F., Kent, D., et al. (2023) · Medical decision making · doi:10.1177/0272989x231152852 · Cited by 5 · see citing works
Self-citationA Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools
Patel, H., Crusco, S., Hansen, D., Morales, J., Patale, T., Chiuzan, C., Sands, M., Naidich, J., Myers, K., Davidson, K. W., et al. (2026) · Journal of General Internal Medicine · doi:10.1007/s11606-026-10381-y · Cited by 4 · see citing works
Self-citationMachine learning to predict completion of treatment for pancreatic cancer
Pasha, S. A., Khalid, A., Levy, T. J., Demyan, L., Hartman, S., Newman, E., Weiss, M. J., King, D. A., Zanos, T. P., & Melis, M. (2024) · Journal of Surgical Oncology · doi:10.1002/jso.27812 · Cited by 4 · see citing works
Self-citationHo, M., Levy, T. J., Koulas, I., Founta, K., Coppa, K., Hirsch, J. S., Davidson, K. W., Spyropoulos, A., & Zanos, T. P. (2023) · Int. J. Medical Informatics · doi:10.1016/j.ijmedinf.2023.105286 · Cited by 4 · see citing works
Self-citationPark, H., Kim, D., You, S. C., Jang, E., Yu, H. T., Kim, T., Kim, D., Sung, J., Pak, H., Lee, M., et al. (2024) · Journal of the American Heart Association · doi:10.1161/jaha.123.032831 · Cited by 4 · see citing works
“17,18 All of the analyses in this study were carried out with the recalibrated European-SCORE2 and US-PCE models.”
Levy, T. J., Makhnevich, A., Barish, M., Zanos, T. P., & Cohen, S. (2023) · Clinical imaging · doi:10.1016/j.clinimag.2023.03.007 · Cited by 2
Self-citationPrediction of intrapartum fever using continuously monitored vital signs and heart rate variability
Debnath, S., Koppel, R., Saadi, N., Potak, D., Weinberger, B., & Zanos, T. P. (2023) · Digital Health · doi:10.1177/20552076231187594 · Cited by 2 · see citing works
Self-citationNavarrete-Campos, J., Pradenas, L., Parada, V., & Scherer, R. F. (2026) · Applied Sciences · doi:10.3390/app16199419 · Cited by 1 · see citing works
Park, H., Kim, D., Jang, E., Yu, H., Kim, T.-H., Uhm, J., Sung, J.-H., Pak, H., Lee, M.-H., Yang, P.-S., et al. (2025) · Journal of the American Heart Association : Cardiovascular and Cerebrovascular Disease · doi:10.1161/jaha.124.037145 · Cited by 1
“21,22 All of the analyses in this study were carried out with the recalibrated CHARGE-AF model (CHARGE-AF refit ).”
Rank-Constrained Adaptation for Reliable Real-World Performance
Gourabathina, A., Jeong, H., Bergamaschi, T. S., Ghassemi, M., & Stultz, C. M. (2026) · doi:10.48550/arxiv.2602.06924
“For example, during the COVID-19 pandemic, clinically meaningful subgroups changed with variants [50], vaccination [85], prior infection [2], and treatment practices [49].”
- Authors
- Levy, T. J., Coppa, K., Cang, J., Barnaby, D. P., Paradis, M. D., Cohen, S. L., Makhnevich, A., van Klaveren, D., Kent, D. M., Davidson, K. W., Hirsch, J. S., & Zanos, T. P.
- Affiliations
- Northwell Health (US)
- Tufts Medical Center (US)
- Corresponding author
- Zanos, Theodoros P.
- ORCID
- Venue
- Nature Communications
- Published
- 2022-11-10
- Volume
- 13
- Issue
- 1
- Pages
- 6812-6812
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC9648888
- OpenAlex
- W4309077223
- Cited by
- 14
- References
- 80
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 5 · 2023: 6
- FWCI
- 0.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Nature Communications (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (acceptedVersion, cc-by)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. Department of Health & Human Services | NIH | National Institute on Aging (R24AG064191)
- U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine (R01LM012836)
- U.S. Department of Health & Human Services | NIH | National Heart, Lung, and Blood Institute (K23HL145114)
- Patient-Centered Outcomes Research Institute (ME-1606-35555)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Infectious Diseases › COVID-19 Clinical Research Studies
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- COVID-19 pandemic, prognostic model, model drift, model updating, self-monitoring, hospitalized patients, model calibration, model discrimination, electronic health records, net benefit, SARS-CoV-2 variants
- MeSH terms
- COVID-19, Calibration, Humans, Prognosis, Retrospective Studies, Cohort Studies, Pandemics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
