Prognostication of patients with COVID-19 using artificial intelligence based on chest x-rays and clinical data: a retrospective study
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Chest x-ray is a relatively accessible, inexpensive, fast imaging modality that might be valuable in the prognostication of patients with COVID-19. We aimed to develop and evaluate an artificial intelligence system using chest x-rays and clinical data to predict disease severity and progression in patients with COVID-19. METHODS: We did a retrospective study in multiple hospitals in the University of Pennsylvania Health System in Philadelphia, PA, USA, and Brown University affiliated hospitals in Providence, RI, USA. Patients who presented to a hospital in the University of Pennsylvania Health System via the emergency department, with a diagnosis of COVID-19 confirmed by RT-PCR and with an available chest x-ray from their initial presentation or admission, were retrospectively identified and randomly divided into training, validation, and test sets (7:1:2). Using the chest x-rays as input to an EfficientNet deep neural network and clinical data, models were trained to predict the binary outcome of disease severity (ie, critical or non-critical). The deep-learning features extracted from the model and clinical data were used to build time-to-event models to predict the risk of disease progression. The models were externally tested on patients who presented to an independent multicentre institution, Brown University affiliated hospitals, and compared with severity scores provided by radiologists. FINDINGS: 1834 patients who presented via the University of Pennsylvania Health System between March 9 and July 20, 2020, were identified and assigned to the model training (n=1285), validation (n=183), or testing (n=366) sets. 475 patients who presented via the Brown University affiliated hospitals between March 1 and July 18, 2020, were identified for external testing of the models. When chest x-rays were added to clinical data for severity prediction, area under the receiver operating characteristic curve (ROC-AUC) increased from 0·821 (95% CI 0·796-0·828) to 0·846 (0·815-0·852; p<0·0001) on internal testing and 0·731 (0·712-0·738) to 0·792 (0·780-0 ·803; p<0·0001) on external testing. When deep-learning features were added to clinical data for progression prediction, the concordance index (C-index) increased from 0·769 (0·755-0·786) to 0·805 (0·800-0·820; p<0·0001) on internal testing and 0·707 (0·695-0·729) to 0·752 (0·739-0·764; p<0·0001) on external testing. The image and clinical data combined model had significantly better prognostic performance than combined severity scores and clinical data on internal testing (C-index 0·805 vs 0·781; p=0·0002) and external testing (C-index 0·752 vs 0·715; p<0·0001). INTERPRETATION: In patients with COVID-19, artificial intelligence based on chest x-rays had better prognostic performance than clinical data or radiologist-derived severity scores. Using artificial intelligence, chest x-rays can augment clinical data in predicting the risk of progression to critical illness in patients with COVID-19. FUNDING: Brown University, Amazon Web Services Diagnostic Development Initiative, Radiological Society of North America, National Cancer Institute and National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
First case of 2019 novel coronavirus in the United States
Holshue, M. L., DeBolt, C., Lindquist, S., Lofy, K. H., Wiesman, J., Bruce, H., Spitters, C., Ericson, K., Wilkerson, S., Tural, A., et al. (2020) · New England Journal of Medicine · doi:10.1056/nejmoa2001191 · Cited by 6,030
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309
Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions
Agresti, A., & Coull, B. A. (1998) · The American Statistician · doi:10.1080/00031305.1998.10480550 · Cited by 3,223
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · cited 1×
“Although a lower performance on external testing is a common finding in deep-learning studies using multi-institutional data, 26,27 including our study, the addition of chest x-rays to clinical data improved the artificial performance on both internal and external testing.”
BackgroundLiang, W., Liang, H., Ou, L., Chen, B., Chen, A., Li, C., Li, Y., Guan, W., Sang, L., Lu, J., et al. (2020) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2020.2033 · Cited by 1,387 · cited 2×
“11,12 The vital signs and laboratory values were taken at the initial time of presentation.”
BackgroundMethodZhang, K., Liu, X., Shen, J., Li, Z., Sang, Y., Wu, X., Zha, Y., Liang, W., Wang, C., Wang, K., et al. (2020) · Cell · doi:10.1016/j.cell.2020.04.045 · Cited by 917 · cited 1×
“…utilised quantitative lung lesion features from chest CT scans of 442 patients with COVID-19 to deter mine prognosis from the time of admission, 25 the model in our study extracted deep-learning features from chest x-rays and measured the time interval from chest x-ray to critical event in…”
MethodSelf-citationBai, H. X., Wang, R., Xiong, Z., Hsieh, B., Chang, K., Halsey, K., Tran, T. M. L., Choi, J. W., Wang, D.-C., Shi, L.-B., et al. (2020) · Radiology · doi:10.1148/radiol.2020201491 · Cited by 381
Self-citationDeep learning for segmentation of brain tumors: Impact of cross‐institutional training and testing
AlBadawy, E. A., Saha, A., & Mazurowski, M. A. (2018) · Medical Physics · doi:10.1002/mp.12752 · Cited by 208
Research works printed but not linked: 16
- Bai, 2020, Performance of radiologists in differentiating COVID-19 from non-COVID-19 viral pneumonia at chest CT, Radiology, 296, e46, 10.1148/radiol.2020200823 · DOI · Paper page
- Yang, 2020, The role of imaging in 2019 novel coronavirus pneumonia (COVID-19), Eur Radiol, 30, 4874, 10.1007/s00330-020-06827-4 · DOI · Paper page
- Chau, 2004, Value of initial chest radiographs for predicting clinical outcomes in patients with severe acute respiratory syndrome, Am J Med, 117, 249, 10.1016/j.amjmed.2004.03.020 · DOI · Paper page
- Toussie, 2020, Clinical and chest radiography features determine patient outcomes in young and middle-aged adults with COVID-19, Radiology, 297, e197, 10.1148/radiol.2020201754 · DOI · Paper page
- Cozzi, 2020, Chest x-ray in new coronavirus disease 2019 (COVID-19) infection: findings and correlation with clinical outcome, Radiol Med (Torino), 125, 730, 10.1007/s11547-020-01232-9 · DOI · Paper page
- Kim, 2020, The role of initial chest x-ray in triaging patients with suspected COVID-19 during the pandemic, Emerg Radiol, 27, 617, 10.1007/s10140-020-01808-y · DOI
- Li, 2020, Automated assessment of COVID-19 pulmonary disease severity on chest radiographs using convolutional Siamese neural networks, medRxiv
- Khuzani, 2020, COVID-classifier: an automated machine learning model to assist in the diagnosis of COVID-19 infection in chest x-ray images, medRxiv
- Kermali, 2020, The role of biomarkers in diagnosis of COVID-19—a systematic review, Life Sci, 254, 10.1016/j.lfs.2020.117788 · DOI · Paper page
- Harrell, 1996, Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors, Stat Med, 15, 361, 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4 · doi:10.1002/(sici)1097-0258(19960229)15:4<361::aid-sim168>3.0.co;2-4
- Heagerty, 2005, Survival model predictive accuracy and ROC curves, Biometrics, 61, 92, 10.1111/j.0006-341X.2005.030814.x · DOI · Paper page
- Cai, 2006, The sensitivity and specificity of markers for event times, Biostatistics, 7, 182, 10.1093/biostatistics/kxi047 · DOI
- Sun, 2020, Lower mortality of COVID-19 by early recognition and intervention: experience from Jiangsu Province, Ann Intensive Care, 10, 33, 10.1186/s13613-020-00650-2 · DOI · Paper page
- Goyal, 2020, Early intervention likely improves mortality in COVID-19 infection, Clin Med (Lond), 20, 248, 10.7861/clinmed.2020-0214 · DOI
- Chassagnon, 2021, AI-driven quantification, staging and outcome prediction of COVID-19 pneumonia, Med Image Anal, 67, 10.1016/j.media.2020.101860 · DOI · Paper page
- Vancheri, 2020, Radiographic findings in 240 patients with COVID-19 pneumonia: time-dependence after the onset of symptoms, Eur Radiol, 30, 6161, 10.1007/s00330-020-06967-7 · DOI
Cited by
Leichtmann, B., Humer, C., Hinterreiter, A., Streit, M., & Mara, M. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107539 · Cited by 200
Soltan, A. A. S., Yang, J., Pattanshetty, R., Novak, A., Yang, Y., Rohanian, O., Beer, S., Soltan, M. A., Thickett, D. R., Fairhead, R., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00272-7 · Cited by 54
21st Century critical care medicine: an overview
Padte, S., Samala Venkata, V., Mehta, P., Tawfeeq, S., Kashyap, R., & Surani, S. (2024) · World Journal of Critical Care Medicine · doi:10.5492/wjccm.v13.i1.90176 · Cited by 26
Cui, X., Chang, Y., Yang, C., Cong, Z., Wang, B., & Leng, Y. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm13010050 · Cited by 19
“Jiao et al. developed an AI system to predict the prognosis of patients with COVID-19 based on chest X-rays [33].”
BackgroundMatsumoto, T., Walston, S. L., Walston, M., Kabata, D., Miki, Y., Shiba, M., & Ueda, D. (2022) · Journal of Digital Imaging · doi:10.1007/s10278-022-00691-y · Cited by 15
“On the other hand, one study showed a CNN and a random survival forest-based model that predicts death or discharge, with time to event for COVID-19 patients [44].”
BackgroundResult
Metadata
- Authors
- Jiao, Z., Choi, J. W., Halsey, K., Tran, T. M. L., Hsieh, B., Wang, D., Eweje, F., Wang, R., Chang, K., Wu, J., Collins, S. A., Yi, T. Y., Delworth, A. T., Liu, T., Healey, T. T., Lu, S., Wang, J., Feng, X., Atalay, M. K., Yang, L., Feldman, M., Zhang, P. J. L., Liao, W.-H., Fan, Y., & Bai, H. X.
- Venue
- The Lancet Digital Health
- Publisher
- Elsevier BV
- Published
- 2021-05
- Volume
- 3
- Issue
- 5
- Pages
- e286-e294
- ISSN
- 2589-7500 (ISSN-L)
- OpenAlex
- W3138653062
- Cited by
- 116
- References
- 10
- FWCI
- 11.90
- Type
- journal article
- Language
- en
- Update notices
- erratum (2021-05-01)
- Open access
- gold
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0/ (vor)
- Funders
- National Cancer Institute (F30CA239407, R03CA249554)
- Brown University
- Radiological Society of North America
- National Institutes of Health (EB022573, DK117297, MH120811, CA223358)
- National Institute of Biomedical Imaging and Bioengineering (5T32EB1680, U54GM115677)
- National Institute of General Medical Sciences
- Amazon Web Services Inc
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Prognostication of patients with COVID-19 using artificial intelligence based on chest x-rays and clinical data: a retrospective study. https://app.cochat.ai/papers/10.1016/s2589-7500(21)00039-x
- The paper
- Jiao, Z., Choi, J. W., Halsey, K., Tran, T. M. L., Hsieh, B., Wang, D., Eweje, F., Wang, R., Chang, K., Wu, J., Collins, S. A., Yi, T. Y., Delworth, A. T., Liu, T., Healey, T. T., Lu, S., Wang, J., Feng, X., Atalay, M. K., . . . Bai, H. X. (2021). Prognostication of patients with COVID-19 using artificial intelligence based on chest x-rays and clinical data: a retrospective study. The Lancet Digital Health. https://doi.org/10.1016/s2589-7500(21)00039-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.