Artificial intelligence-based model for COVID-19 prognosis incorporating chest radiographs and clinical data; a retrospective model development and validation study
Open accessAbstract
Objectives: The purpose of this study was to develop an artificial intelligence-based model to prognosticate COVID-19 patients at admission by combining clinical data and chest radiographs. Methods: This retrospective study used the Stony Brook University COVID-19 dataset of 1384 inpatients. After exclusions, 1356 patients were randomly divided into training (1083) and test datasets (273). We implemented three artificial intelligence models, which classified mortality, ICU admission, or ventilation risk. Each model had three submodels with different inputs: clinical data, chest radiographs, and both. We showed the importance of the variables using SHapley Additive exPlanations (SHAP) values. Results: The mortality prediction model was best overall with area under the curve, sensitivity, specificity, and accuracy of 0.79 (0.72–0.86), 0.74 (0.68–0.79), 0.77 (0.61–0.88), and 0.74 (0.69–0.79) for the clinical data-based model; 0.77 (0.69–0.85), 0.67 (0.61–0.73), 0.81 (0.67–0.92), 0.70 (0.64–0.75) for the image-based model, and 0.86 (0.81–0.91), 0.76 (0.70–0.81), 0.77 (0.61–0.88), 0.76 (0.70–0.81) for the mixed model. The mixed model had the best performance (p value < 0.05). The radiographs ranked fourth for prognostication overall, and first of the inpatient tests assessed. Conclusions: These results suggest that prognosis models become more accurate if AI-derived chest radiograph features and clinical data are used together. Advances in knowledge: This AI model evaluates chest radiographs together with clinical data in order to classify patients as having high or low mortality risk. This work shows that chest radiographs taken at admission have significant COVID-19 prognostic information compared to clinical data other than age and sex.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The COVID-19 pandemic increased the number of COVID-19-related studies from Japan, with a significant increase in such research focusing on AI.
“Furthermore, owing to the coronavirus disease 2019 (COVID-19) pandemic, the number of COVID-19-related studies from Japan has increased [54–61], with a significant increase in such research focusing on AI [62–64].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [62]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
The third international consensus definitions for sepsis and septic shock (Sepsis-3).
Singer, M., Deutschman, C. S., Seymour, C. W., Shankar-Hari, M., Annane, D., Bauer, M., Bellomo, R., Bernard, G. R., Chiche, J.-D., Coopersmith, C. M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0287 · Cited by 28,250 · see citing works
The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository
Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., et al. (2013) · Journal of Digital Imaging · doi:10.1007/s10278-013-9622-7 · Cited by 4,599 · see citing works
Proceedings of the 21st International Conference on Neural Information Processing Systems
Koller, D., Schuurmans, D., Bengio, Y., & Bottou, L. (2008) · Cited by 4,461 · see citing works
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
Li, L., Qin, L., Xu, Z., Yin, Y., Wang, X., Kong, B., Bai, J., Lu, Y., Fang, Z., Song, Q., et al. (2020) · Radiology · doi:10.1148/radiol.2020200905 · Cited by 1,927 · see citing works
RECIST 1.1—Update and clarification: From the RECIST committee
Schwartz, L. H., Litière, S., de Vries, E., Ford, R., Gwyther, S., Mandrekar, S., Shankar, L., Bogaerts, J., Chen, A., Dancey, J., et al. (2016) · European Journal of Cancer · doi:10.1016/j.ejca.2016.03.081 · Cited by 1,906 · see citing works
Frequency and Distribution of Chest Radiographic Findings in Patients Positive for COVID-19
Wong, H. Y. F., Lam, H. Y. S., Fong, A. H.-T., Leung, S. T., Chin, T. W.-Y., Lo, C. S.-Y., Lui, M. M., Lee, J. C. Y., Chiu, K. W.-H., Chung, T. W., et al. (2020) · Radiology · doi:10.1148/radiol.2020201160 · Cited by 1,485 · see citing works
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443 · see citing works
Liang, W., Liang, H., Ou, L., Chen, B., Chen, A., Li, C., Li, Y., Guan, W., Sang, L., Lu, J., et al. (2020) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2020.2033 · Cited by 1,387 · see citing works
Federated learning for predicting clinical outcomes in patients with COVID-19
Dayan, I., Roth, H. R., Zhong, A., Harouni, A., Gentili, A., Abidin, A. Z., Liu, A., Costa, A. B., Wood, B. J., Tsai, C.-S., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01506-3 · Cited by 842 · see citing works
Deep Learning—A Technology With the Potential to Transform Health Care
Hinton, G. (2018) · JAMA · doi:10.1001/jama.2018.11100 · Cited by 667 · see citing works
Izcovich, A., Ragusa, M. A., Tortosa, F. G., Marzio, M. A. L., Agnoletti, C., Bengolea, A., Ceirano, A., Espinosa, F., Saavedra, E., Sanguine, V., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0241955 · Cited by 658 · see citing works
On the interpretability of artificial intelligence in radiology: challenges and opportunities
Reyes, M., Meier, R., Pereira, S., Silva, C. A., Dahlweid, F.-M., Tengg-Kobligk, H. V., Summers, R. M., & Wiest, R. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020190043 · Cited by 484 · see citing works
Well-aerated Lung on Admitting Chest CT to Predict Adverse Outcome in COVID-19 Pneumonia
Colombi, D., Bodini, F. C., Petrini, M., Maffi, G., Morelli, N., Milanese, G., Silva, M., Sverzellati, N., & Michieletti, E. (2020) · Radiology · doi:10.1148/radiol.2020201433 · Cited by 456 · see citing works
Utilization of machine-learning models to accurately predict the risk for critical COVID-19
Assaf, D., Gutman, Y., Neuman, Y., Segal, G., Amit, S., Gefen-Halevi, S., Shilo, N., Epstein, A., Mor-Cohen, R., Biber, A., et al. (2020) · Internal and Emergency Medicine · doi:10.1007/s11739-020-02475-0 · Cited by 275 · see citing works
Early triage of critically ill COVID-19 patients using deep learning
Liang, W., Yao, J., Chen, A., Lv, Q., Zanin, M., Liu, J., Wong, S., Li, Y., Lu, J., Liang, H., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-17280-8 · Cited by 270 · see citing works
Wu, G., Yang, P., Woodruff, H. C., Rao, X., Guiot, J., Frix, A.-N., Moutschen, M. P., Louis, R. E., Li, J., Li, J., et al. (2020) · European Respiratory Journal · doi:10.1183/13993003.01104-2020 · Cited by 243 · see citing works
Severity assessment of COVID-19 using CT image features and laboratory indices
Tang, Z., Zhao, W., Xie, X., Zhong, Z., Shi, F., Ma, T., Liu, J., & Shen, D. G. (2020) · Physics in Medicine and Biology · doi:10.1088/1361-6560/abbf9e · Cited by 242 · see citing works
Using Machine Learning to Predict ICU Transfer in Hospitalized COVID-19 Patients
Cheng, F.-Y., Joshi, H. N., Tandon, P., Freeman, R., Reich, D. L., Mazumdar, M., Kohli-Seth, R. D., Levin, M. A., Timsina, P., & Kia, A. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9061668 · Cited by 197 · see citing works
Gupta, R. K., Marks, M., Samuels, T. H. A., Luintel, A., Rampling, T., Chowdhury, H., Quartagno, M., Nair, A., Lipman, M. C. I., Abubakar, I., et al. (2020) · European Respiratory Journal · doi:10.1183/13993003.03498-2020 · Cited by 189 · see citing works
Lassau, N., Ammari, S., Chouzenoux, É., Gortais, H., Hérent, P., Devilder, M., Soliman, S., Meyrignac, O., Talabard, M.-P., Lamarque, J.-P., et al. (2021) · Nature Communications · doi:10.1038/s41467-020-20657-4 · Cited by 187 · see citing works
Piva, S., Filippini, M., Turla, F., et al. (2020) · Journal of Critical Care · doi:10.1016/j.jcrc.2020.04.004 · Cited by 178 · see citing works
Shamout, F. E., Shen, Y., Wu, N., Kaku, A., Park, J., Makino, T., Jastrzȩbski, S., Witowski, J. S., Wang, D., Zhang, B., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00453-0 · Cited by 153 · see citing works
Luo, M., Liu, J., Jiang, W., Yue, S., Liu, H., & Wei, S. (2020) · JCI Insight · doi:10.1172/jci.insight.139024 · Cited by 130 · see citing works
Mushtaq, J., Pennella, R., Lavalle, S., Colarieti, A., Steidler, S., Martinenghi, C. M. A., Palumbo, D., Esposito, A., Rovere-Querini, P., Tresoldi, M., et al. (2021) · European Radiology · doi:10.1007/s00330-020-07269-8 · Cited by 124 · see citing works
Jiao, Z., Choi, J. W., Halsey, K., Tran, T. M. L., Hsieh, B., Wang, D., Eweje, F. R., Wang, R., Chang, K., Wu, J., et al. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00039-x · Cited by 117 · see citing works
Gidari, A., Socio, G. V. D., Sabbatini, S., & Francisci, D. (2020) · Scandinavian Journal of Infectious Diseases · doi:10.1080/23744235.2020.1784457 · Cited by 109 · see citing works
Soda, P., D’Amico, N. C., Tessadori, J., Valbusa, G., Guarrasi, V., Bortolotto, C., Akbar, M. U., Sicilia, R., Cordelli, E., Fazzini, D., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102216 · Cited by 92 · see citing works
Rapid identification of COVID-19 severity in CT scans through classification of deep features
Yu, Z., Li, X., Sun, H., et al. (2020) · BioMedical Engineering OnLine · doi:10.1186/s12938-020-00807-x · Cited by 78 · see citing works
Artificial intelligence for prediction of COVID-19 progression using CT imaging and clinical data
Wang, R., Jiao, Z., Yang, L., Choi, J. W., Xiong, Z., Halsey, K., Tran, T. M. L., Pan, I., Collins, S., Feng, X., et al. (2021) · European Radiology · doi:10.1007/s00330-021-08049-8 · Cited by 67 · see citing works
Development and validation of prognosis model of mortality risk in patients with COVID-19
Ma, X., Ng, M., Xu, S. B., Xu, Z., Qiu, H., Liu, Y., Lyu, J., You, J., Zhao, P., Wang, S., et al. (2020) · Epidemiology and Infection · doi:10.1017/s0950268820001727 · Cited by 59 · see citing works
Kwon, Y. J., Toussie, D., Finkelstein, M. S., Cedillo, M. A., Maron, S. Z., Manna, S., Voutsinas, N., Eber, C. D., Jacobi, A., Bernheim, A., et al. (2020) · Radiology Artificial Intelligence · doi:10.1148/ryai.2020200098 · Cited by 53 · see citing works
AI-based analysis of CT images for rapid triage of COVID-19 patients
Xu, Q., Zhan, X., Zhou, Z., Li, Y., Xie, P., Zhang, S., Li, X., Yu, Y., Zhou, C., Zhang, L. J., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00446-z · Cited by 42 · see citing works
Kim, C. K., Choi, J. W., Jiao, Z., Wang, D., Wu, J., Yi, T. Y., Halsey, K., Eweje, F. R., Tran, T. M. L., Liu, C., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00546-w · Cited by 36 · see citing works
Rodriguez, V., Bhave, S., Chen, R., Pang, C., Hripcsak, G. M., Sengupta, S., Elhadad, N., Green, R., Adelman, J. S., Schlosser, K. R., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab029 · Cited by 26 · see citing works
Chamberlin, J., Aquino, G. J., Nance, S., Wortham, A., Leaphart, N., Paladugu, N., Brady, S., Baird, H., Fiegel, M. J., Fitzpatrick, L., et al. (2022) · BMC Infectious Diseases · doi:10.1186/s12879-022-07617-7 · Cited by 24 · see citing works
Iori, M., Castelnuovo, C. D., Verzellesi, L., Meglioli, G., Lippolis, D. G., Nitrosi, A., Monelli, F., Besutti, G., Trojani, V., Bertolini, M., et al. (2022) · Applied Sciences · doi:10.3390/app12083903 · Cited by 20 · see citing works
Diagnostic imaging in COVID-19 pneumonia: a literature review
Campagnano, S., Angelini, F., Fonsi, G. B., Novelli, S., & Drudi, F. M. (2021) · Journal of Ultrasound · doi:10.1007/s40477-021-00559-x · Cited by 20 · see citing works
Hiremath, A., Bera, K., Yuan, L., et al. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3103389 · Cited by 20 · see citing works
Barry, M. A., Alotaibi, M., Almohaya, A. M., Aldrees, A., Alhijji, A., Al-Thabit, N., Alhasani, S., Akkielah, L., AlRajhi, A., Nouh, T. A., et al. (2021) · Journal of Infection and Public Health · doi:10.1016/j.jiph.2021.09.023 · Cited by 17 · see citing works
Matsumoto, T., Walston, S. L., Walston, M., Kabata, D., Miki, Y., Shiba, M., & Ueda, D. (2022) · Journal of Digital Imaging · doi:10.1007/s10278-022-00691-y · Cited by 15 · see citing works
Self-citationCalvillo-Batllés, P., Cerdá-Alberich, L., Fonfría-Esparcia, C., Carreres-Ortega, A., Muñoz-Núñez, C. F., Trilles-Olaso, L., & Martí‐Bonmatí, L. (2022) · Radiología (English Edition) · doi:10.1016/j.rxeng.2021.09.004 · Cited by 12 · see citing works
Gourdeau, D., Potvin, O., Biem, J. H., Cloutier, F., Abrougui, L., Archambault, P. M., Chartrand‐Lefebvre, C., Dieumegarde, L., Gagné, C., Gagnon, L., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10136-9 · Cited by 11 · see citing works
Teaching an old dog new tricks: The prognostic role of CACS in hospitalized COVID-19 patients
Ahmed, A. I., Raggi, P., & Al-Mallah, M. H. (2021) · Atherosclerosis · doi:10.1016/j.atherosclerosis.2021.05.015 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Paszke A, Gross S, Massa F, et al. Pytorch: an imperative style, high-performance deep learning library. Adv Neural Inf Process Syst 2019; 32: 8026-37.
- Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In: Paper presented at the In: Paper presented at the 2017 IEEE International Conference on Computer Vision (ICCV), Venice. https:// doi.org/10.1109/ICCV.2017.74 · DOI · Paper page
- Sobin LH. TNM Online. 8th edition. Hoboken, NJ: Wiley-Blackwell; 2016. https:// doi.org/10.1002/9780471420194 · DOI
Cited by
Sorted by citations
Fairness of artificial intelligence in healthcare: review and recommendations
Ueda, D., Kakinuma, T., Fujita, S., Kamagata, K., Fushimi, Y., Ito, R., Matsui, Y., Nozaki, T., Nakaura, T., Fujima, N., et al. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01474-3 · Cited by 607 · see citing works
Self-citation
- Authors
- Walston, S. L., Matsumoto, T., Miki, Y., & Ueda, D.
- Affiliations
- Osaka City University (JP)
- Osaka Metropolitan University (JP)
- Venue
- The British Journal of Radiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2022-12-01
- Volume
- 95
- Issue
- 1140
- Pages
- 20220058-20220058
- ISSN
- 0007-1285 (ISSN-L)
- 1748-880X
- PMC
- PMC9733620
- OpenAlex
- W4301392540
- Cited by
- 9
- References
- 49
- Citations by year
- 2025: 1 · 2024: 6 · 2023: 2
- FWCI
- 1.28
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- British Journal of Radiology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Institutional Repositories DataBase (IRDB) (submittedVersion, cc-by-nc-nd)
- Funders
- Japan Society for the Promotion of Science (17K10415)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Infectious Diseases › COVID-19 Clinical Research Studies
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- COVID-19 pandemic, prognosis, chest radiograph, mortality prediction, intensive care unit admission, clinical data, artificial intelligence, SHAP, multimodal prediction model
- MeSH terms
- COVID-19, Artificial Intelligence, Humans, Prognosis, Radiography, Retrospective Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
