Development and validation of a multimodal-based prognosis and intervention prediction model for COVID-19 patients in a multicenter cohort
Open accessAbstract
The ability to accurately predict the prognosis and intervention requirements for treating highly infectious diseases, such as COVID-19, can greatly support the effective management of patients, especially in resource-limited settings. The aim of the study is to develop and validate a multimodal artificial intelligence (AI) system using clinical findings, laboratory data and AI-interpreted features of chest X-rays (CXRs), and to predict the prognosis and the required interventions for patients diagnosed with COVID-19, using multi-center data. In total, 2282 real-time reverse transcriptase polymerase chain reaction-confirmed COVID-19 patients’ initial clinical findings, laboratory data and CXRs were retrospectively collected from 13 medical centers in South Korea, between January 2020 and June 2021. The prognostic outcomes collected included intensive care unit (ICU) admission and in-hospital mortality. Intervention outcomes included the use of oxygen (O2) supplementation, mechanical ventilation and extracorporeal membrane oxygenation (ECMO). A deep learning algorithm detecting 10 common CXR abnormalities (DLAD-10) was used to infer the initial CXR taken. A random forest model with a quantile classifier was used to predict the prognostic and intervention outcomes, using multimodal data. The area under the receiver operating curve (AUROC) values for the single-modal model, using clinical findings, laboratory data and the outputs from DLAD-10, were 0.742 (95% confidence interval [CI], 0.696−0.788), 0.794 (0.745−0.843) and 0.770 (0.724−0.815), respectively. The AUROC of the combined model, using clinical findings, laboratory data and DLAD-10 outputs, was significantly higher at 0.854 (0.820−0.889) than that of all other models (p < 0.001, using DeLong’s test). In the order of importance, age, dyspnea, consolidation and fever were significant clinical variables for prediction. The most predictive DLAD-10 output was consolidation. We have shown that a multimodal AI model can improve the performance of predicting both the prognosis and intervention in COVID-19 patients, and this could assist in effective treatment and subsequent resource management. Further, image feature extraction using an established AI engine with well-defined clinical outputs, and combining them with different modes of clinical data, could be a useful way of creating an understandable multimodal prediction model.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Retrospective validation on 2282 COVID-19 patients from 13 medical centers demonstrated that the Lee et al. (2022) multimodal model significantly outperformed the unimodal imaging model with an AUC of 0.854 vs. 0.770.
“Retrospective validation on 2282 COVID-19 patients from 13 medical centers demonstrated that the multimodal model significantly outperformed the unimodal imaging model with an AUC of 0.854 vs. 0.770, respectively.”
From Navigating the landscape of multimodal AI in medicine: A scoping review on technical challenges and clinical applications (2025) · cites this paper as Lee et al., 2022
Results §4.2 reports that the multimodal model significantly outperformed the CXR-based model for predicting all adverse events (AUROC 0.854 vs. 0.770; p < 0.001). The study was retrospective, but Results §4.1 says 2282 patients from 13 centers comprised the full dataset; the held-out validation set contained 551 patients from 5 centers, not 2282 patients. (LLM, conf=0.88)
Experimental — accuracy not yet measuredA feature importance analysis revealed that clinical variables such as age and dyspnea played an essential role in the Lee et al. (2022) multimodal model's prediction.
“A feature importance analysis revealed that, among others, clinical variables such as age and dyspnea played an essential role in the multimodal model's prediction.”
From Navigating the landscape of multimodal AI in medicine: A scoping review on technical challenges and clinical applications (2025) · cites this paper as Lee et al., 2022
Lee et al. (2022) focused on developing and validating a multimodal prognosis and treatment selection model in COVID-19 patients.
“Second, Lee et al. (2022) focused on developing and validating a multimodal prognosis and treatment selection model in COVID-19 patients.”
From Navigating the landscape of multimodal AI in medicine: A scoping review on technical challenges and clinical applications (2025) · cites this paper as Lee et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · see citing works
pROC: an open-source package for R and S+ to analyze and compare ROC curves
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-77 · Cited by 13,697 · see citing works
The APACHE III Prognostic System
Knaus, W. A., Wagner, D. P., Draper, E. A., Zimmerman, J. E., Bergner, M., BASTOS, P. G., Sirio, C., Murphy, D. J., Lotring, T., Damiano, A., et al. (1991) · CHEST Journal · doi:10.1378/chest.100.6.1619 · Cited by 3,524 · see citing works
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
COVID-19 pathophysiology: A review
Yuki, K., Shibamura‐Fujiogi, M., & Koutsogiannaki, S. (2020) · Clinical Immunology · doi:10.1016/j.clim.2020.108427 · Cited by 2,217 · see citing works
Frequency and Distribution of Chest Radiographic Findings in Patients Positive for COVID-19
Wong, H. Y. F., Lam, H. Y. S., Fong, A. H.-T., Leung, S. T., Chin, T. W.-Y., Lo, C. S.-Y., Lui, M. M., Lee, J. C. Y., Chiu, K. W.-H., Chung, T. W., et al. (2020) · Radiology · doi:10.1148/radiol.2020201160 · Cited by 1,485 · see citing works
Self-citationThe Role of Chest Imaging in Patient Management During the COVID-19 Pandemic
Rubin, G. D., Ryerson, C. J., Haramati, L. B., Sverzellati, N., Kanne, J. P., Raoof, S., Schluger, N. W., Volpi, A., Yim, J., Martin, I. B. K., et al. (2020) · CHEST Journal · doi:10.1016/j.chest.2020.04.003 · Cited by 1,249 · see citing works
Rubin, G. D., Ryerson, C. J., Haramati, L. B., Sverzellati, N., Kanne, J. P., Raoof, S., Schluger, N. W., Volpi, A., Yim, J.-J., Martin, I. B. K., et al. (2020) · Radiology · doi:10.1148/radiol.2020201365 · Cited by 1,101 · see citing works
Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00307-0 · Cited by 1,044 · cited 1× · see citing works
“It has been shown that previous imaging-based AI models for COVID-19 prognostication may be inadequate for real clinical usage [35,36].”
MethodRoberts, M. T., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Avilés-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2020) · Apollo (University of Cambridge) · doi:10.17863/cam.64000 · Cited by 902 · see citing works
COVID-19: Immunology and treatment options
Felsenstein, S., Herbert, J. A., McNamara, P. S., & Hedrich, C. M. (2020) · Clinical Immunology · doi:10.1016/j.clim.2020.108448 · Cited by 669 · see citing works
Pesapane, F., Volonté, C., Codari, M., & Sardanelli, F. (2018) · Insights into Imaging · doi:10.1007/s13244-018-0645-y · Cited by 479 · see citing works
Lancet, T. (2020) · The Lancet · doi:10.1016/s0140-6736(20)30938-7 · Cited by 479 · see citing works
Variable selection using Mean Decrease Accuracy and Mean Decrease Gini based on Random Forest
Han, H., Guo, X., & Hua, Y. (2016) · IEEE International Conference on Software Engineering and Service Science (ICSESS) · doi:10.1109/icsess.2016.7883053 · Cited by 448 · see citing works
Hwang, E. J., Park, S., Jin, K.-N., Kim, J. I., Choi, S. Y., Lee, J. H., Goo, J. M., Aum, J., Yim, J.-J., Cohen, J. G., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.1095 · Cited by 447 · see citing works
Self-citationLower mortality of COVID-19 by early recognition and intervention: experience from Jiangsu Province
Sun, Q., Qiu, H., Huang, M., & Yang, Y. (2020) · Annals of Intensive Care · doi:10.1186/s13613-020-00650-2 · Cited by 410 · see citing works
Toussie, D., Voutsinas, N., Finkelstein, M. S., Cedillo, M. A., Manna, S., Maron, S. Z., Jacobi, A., Chung, M., Bernheim, A., Eber, C. D., et al. (2020) · Radiology · doi:10.1148/radiol.2020201754 · Cited by 344 · see citing works
Self-citationA Closer Look Into Global Hospital Beds Capacity and Resource Shortages During the COVID-19 Pandemic
Sen-Crowe, B., Sutherland, M., McKenney, M. G., et al. (2020) · Journal of Surgical Research · doi:10.1016/j.jss.2020.11.062 · Cited by 299 · see citing works
Mass Casualty Triage: An Evaluation of the Data and Development of a Proposed National Guideline
Lerner, E. B., Schwartz, R. B., Coule, P. L., Weinstein, E. S., Cone, D. C., Hunt, R. C., Sasser, S. M., Liu, J. M., Nudell, N. G., Wedmore, I. S., et al. (2008) · Disaster Medicine and Public Health Preparedness · doi:10.1097/dmp.0b013e318182194e · Cited by 277 · see citing works
Systematic review of reviews of symptoms and signs of COVID-19 in children and adolescents
Viner, R., Ward, J., Hudson, L. D., Ashe, M., Patel, S. V., Hargreaves, D. S., & Whittaker, E. (2020) · Archives of Disease in Childhood · doi:10.1136/archdischild-2020-320972 · Cited by 222 · see citing works
COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System
Murphy, K., Smits, H., Knoops, A. J. G., Korst, M. B. J. M., Samson, T., Scholten, E. T., Schalekamp, S., Schaefer‐Prokop, C. M., Philipsen, R. H. H. M., Meijers, A., et al. (2020) · Radiology · doi:10.1148/radiol.2020201874 · Cited by 218 · see citing works
Chest x-ray findings and temporal lung changes in patients with COVID-19 pneumonia
Rousan, L. A., El-Obeid, E., Karrar, M., & Khader, Y. S. (2020) · BMC Pulmonary Medicine · doi:10.1186/s12890-020-01286-5 · Cited by 210 · see citing works
Khuzani, A. Z., Heidari, M., & Shariati, S. A. (2021) · Scientific Reports · doi:10.1038/s41598-021-88807-2 · Cited by 210 · see citing works
Pan, P., Li, Y., Xiao, Y., Han, B., Su, L., Su, M., Li, Y., Zhang, S., Jiang, D., Chen, X., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/23128 · Cited by 128 · see citing works
Jiao, Z., Choi, J. W., Halsey, K., Tran, T. M. L., Hsieh, B., Wang, D., Eweje, F. R., Wang, R., Chang, K., Wu, J., et al. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00039-x · Cited by 117 · see citing works
Clinical Predictive Models for COVID-19: Systematic Study
Schwab, P., Schütte, A. D., Dietz, B., & Bauer, S. (2020) · Journal of Medical Internet Research · doi:10.2196/21439 · Cited by 110 · see citing works
Nam, J. G., Kim, M., Park, J., Hwang, E. J., Lee, J. H., Hong, J. H., Goo, J. M., & Park, C. M. (2021) · European Respiratory Journal · doi:10.1183/13993003.03061-2020 · Cited by 103 · see citing works
Self-citationYoo, H., Lee, S. H., Arru, C. D., Khera, R. D., Singh, R., Siebert, S. M., Kim, D., Lee, Y., Park, J. H., Eom, H. J., et al. (2021) · European Radiology · doi:10.1007/s00330-021-08074-7 · Cited by 86 · see citing works
Lee, J. H., Sun, H. Y., Park, S., Kim, H., Hwang, E. J., Goo, J. M., & Park, C. M. (2020) · Radiology · doi:10.1148/radiol.2020201240 · Cited by 82 · see citing works
Self-citationThe role of initial chest X-ray in triaging patients with suspected COVID-19 during the pandemic
Kim, H. W., Capaccione, K. M., Li, G., Luk, L., Widemon, R. S., Rahman, O., Beylergil, V., Mitchell, R., D’Souza, B. M., Leb, J. S., et al. (2020) · Emergency Radiology · doi:10.1007/s10140-020-01808-y · Cited by 60 · see citing works
Detection of COVID-19 in smartphone-based breathing recordings: A pre-screening deep learning tool
Alkhodari, M., & Khandoker, A. H. (2022) · PLoS ONE · doi:10.1371/journal.pone.0262448 · Cited by 59 · see citing works
Chest x-ray in the COVID-19 pandemic: Radiologists’ real-world reader performance
Cozzi, A., Schiaffino, S., Arpaia, F., Pepa, G. D., Tritella, S., Bertolotti, P., Menicagli, L., Monaco, C. G., Carbonaro, L. A., Spairani, R., et al. (2020) · European Journal of Radiology · doi:10.1016/j.ejrad.2020.109272 · Cited by 54 · see citing works
Hwang, E. J., Kim, H., Yoon, S. H., Goo, J. M., & Park, C. M. (2020) · Korean Journal of Radiology · doi:10.3348/kjr.2020.0536 · Cited by 49 · see citing works
Hwang, E. J., Kim, K. B., Kim, J. Y., Lim, J., Nam, J. G., Choi, H., Kim, H., Yoon, S. H., Goo, J. M., & Park, C. M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252440 · Cited by 48 · see citing works
Self-citationEarly intervention likely improves mortality in COVID-19 infection
Goyal, D., Mansab, F., Iqbal, A., et al. (2020) · Clinical Medicine · doi:10.7861/clinmed.2020-0214 · Cited by 47 · see citing works
Li, M., & Chapman, G. B. (2020) · The Wiley Encyclopedia of Health Psychology · doi:10.1002/9781119057840.ch84 · Cited by 45 · see citing works
Hwang, E. J., Hong, J. H., Lee, K. H., Kim, J. I., Nam, J. G., Kim, D., Choi, H., Yoo, S., Goo, J. M., & Park, C. M. (2020) · European Radiology · doi:10.1007/s00330-020-06771-3 · Cited by 44 · see citing works
Self-citationGong, K., Wu, D., Arru, C. D., Homayounieh, F., Neumark, N., Guan, J., Buch, V., Kim, K., Bizzo, B. C., Ren, H., et al. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.109583 · Cited by 39 · see citing works
Chau, T., Lee, P.-O., Choi, K., Lee, C.-M., Ma, K.-F., Tsang, T. Y. O., Tso, Y.-K., Chiu, M. C., Tong, W., Yu, W. C., et al. (2004) · The American Journal of Medicine · doi:10.1016/j.amjmed.2004.03.020 · Cited by 38 · see citing works
Critical medication shortages further dwindling hospital resources during COVID-19
Farrell, N. M., Hayes, B. D., & Linden, J. A. (2020) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2020.05.059 · Cited by 29 · see citing works
Kim, E. Y., Kim, Y. J., Choi, W.-J., Lee, G. P., Choi, Y. R., Jin, K. N., & Cho, Y. J. (2021) · PLoS ONE · doi:10.1371/journal.pone.0246472 · Cited by 24 · see citing works
Self-citationLópez-Cabrera, J. D., Orozco‐Morales, R., Portal-Díaz, J. A., Enríquez, O. A. L., & Pérez-Díaz, M. (2021) · Health and Technology · doi:10.1007/s12553-021-00609-8 · Cited by 24 · see citing works
Jang, S. B., Lee, S. H., Lee, D. E., Park, S.-Y., Kim, J. K., Cho, J. W., Cho, J., Kim, K. B., Park, B., Park, J., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0242759 · Cited by 21 · see citing works
Self-citationHow Artificial Intelligence and New Technologies Can Help the Management of the COVID-19 Pandemic
Barbieri, D., Giuliani, E., Prete, A. D., Losi, A., Villani, M., & Barbieri, A. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18147648 · Cited by 20 · see citing works
Establishment of a Nationwide Korean Imaging Cohort of Coronavirus Disease 2019
Yoon, S. H., Ham, S., Nam, B. D., Chae, K. J., Lee, D., Yoo, J. Y., Bak, S. H., Kim, J. Y., Kim, J. H., Kim, K. B., et al. (2020) · Journal of Korean Medical Science · doi:10.3346/jkms.2020.35.e413 · Cited by 14 · see citing works
Self-citationSubclass-based semi-random data partitioning for improving sample representativeness
Liu, H., Chen, S., & Cocea, M. (2018) · Information Sciences · doi:10.1016/j.ins.2018.11.002 · Cited by 13 · see citing works
Monjur, O., Preo, R. B., Shams, A. B., Raihan, M. M. S., & Fairoz, F. (2021) · BioMed · doi:10.3390/biomed1020011 · Cited by 9 · see citing works
Reflections on the Clinical Acceptance of Artificial Intelligence
Schneider, J., & Agus, M. (2021) · Lecture notes in bioengineering · doi:10.1007/978-3-030-67303-1_9 · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [34]Shams A.B. Raihan M.M.S. Khan M.M.U. Preo R.B. Monjur O. Telehealthcare and COVID-19: A Noninvasive & Low Cost Invasive, Scalable and Multimodal Real-Time Smartphone Application for Early Diagnosis of SARS-CoV-2 Infection arXiv 2021 2109.07846
Cited by
Sorted by citations
Schouten, D., Nicoletti, G., Dille, B., Chia, C., Vendittelli, P., Schuurmans, M., Litjens, G., & Khalili, N. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103621 · Cited by 108 · see citing works
- Authors
- Lee, J. H., Ahn, J. S., Chung, M. J., Jeong, Y. J., Kim, J. H., Lim, J. K., Kim, J. Y., Kim, Y. J., Lee, J. E., & Kim, E. Y.
- Affiliations
- Chonnam National University Hospital (KR)
- Lunit (South Korea) (KR)
- Samsung Medical Center (KR)
- Pusan National University Hospital (KR)
- Chungnam National University (KR)
- Kyungpook National University Hospital (KR)
- Gachon University (KR)
- Corresponding author
- Lee, Jong Eun
- Kim, Eun Young
- ORCID
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2022-07-02
- Volume
- 22
- Issue
- 13
- Pages
- 5007
- ISSN
- 1424-8220 (ISSN-L)
- PMC
- PMC9269794
- OpenAlex
- W4283810499
- Cited by
- 9
- References
- 48
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 2 · 2023: 2 · 2022: 1
- FWCI
- 1.14
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Sensors (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- information and Communications Promotion Fund (ICT promotion fund) (FRD2021-11)
- National IT industry Promotion Agency (NIPA) (FRD2021-11)
- Ministry of Science and ICD (MSIT), Republic of Korea (FRD2021-11)
- Gachon University Gil Medical Center (FRD2021-11)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Infectious Diseases › COVID-19 Clinical Research Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- COVID-19 pandemic, outcome prediction, intervention prediction, multimodal AI, chest X-ray abnormalities, deep learning, random forest, intensive care unit admission, in-hospital mortality, mechanical ventilation, extracorporeal membrane oxygenation, supplemental oxygen, laboratory findings, clinical features, resource management, South Korea, multicenter cohort, feature importance, resource-limited settings
- MeSH terms
- COVID-19, Artificial Intelligence, Humans, Intensive Care Units, Prognosis, Retrospective Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
