Artificial Intelligence Augmentation of Radiologist Performance in Distinguishing COVID-19 from Pneumonia of Other Origin at Chest CT
Open accessAbstract
Background Coronavirus disease 2019 (COVID-19) and pneumonia of other diseases share similar CT characteristics, which contributes to the challenges in differentiating them with high accuracy. Purpose To establish and evaluate an artificial intelligence (AI) system for differentiating COVID-19 and other pneumonia at chest CT and assessing radiologist performance without and with AI assistance. Materials and Methods A total of 521 patients with positive reverse transcription polymerase chain reaction results for COVID-19 and abnormal chest CT findings were retrospectively identified from 10 hospitals from January 2020 to April 2020. A total of 665 patients with non–COVID-19 pneumonia and definite evidence of pneumonia at chest CT were retrospectively selected from three hospitals between 2017 and 2019. To classify COVID-19 versus other pneumonia for each patient, abnormal CT slices were input into the EfficientNet B4 deep neural network architecture after lung segmentation, followed by a two-layer fully connected neural network to pool slices together. The final cohort of 1186 patients (132 583 CT slices) was divided into training, validation, and test sets in a 7:2:1 and equal ratio. Independent testing was performed by evaluating model performance in separate hospitals. Studies were blindly reviewed by six radiologists without and then with AI assistance. Results The final model achieved a test accuracy of 96% (95% confidence interval [CI]: 90%, 98%), a sensitivity of 95% (95% CI: 83%, 100%), and a specificity of 96% (95% CI: 88%, 99%) with area under the receiver operating characteristic curve of 0.95 and area under the precision-recall curve of 0.90. On independent testing, this model achieved an accuracy of 87% (95% CI: 82%, 90%), a sensitivity of 89% (95% CI: 81%, 94%), and a specificity of 86% (95% CI: 80%, 90%) with area under the receiver operating characteristic curve of 0.90 and area under the precision-recall curve of 0.87. Assisted by the probabilities of the model, the radiologists achieved a higher average test accuracy (90% vs 85%, Δ = 5, P < .001), sensitivity (88% vs 79%, Δ = 9, P < .001), and specificity (91% vs 88%, Δ = 3, P = .001). Conclusion Artificial intelligence assistance improved radiologists' performance in distinguishing coronavirus disease 2019 pneumonia from non–coronavirus disease 2019 pneumonia at chest CT. © RSNA, 2020 Online supplemental material is available for this article.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Our results run counter to recent suggestions and beliefs that generative AI can promote creativity by having AI generate outputs and people provide the final judgment on the quality of these outputs (i.e., the editor role).
“Our results run counter to recent suggestions and beliefs espoused by technology leaders and scholars that generative AI can promote creativity by having AI generate outputs and people provide the final judgment on the quality of these outputs41–43.”
From Establishing the importance of co-creation and self-efficacy in creative collaboration with artificial intelligence (2024) · cites this paper as [41]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China
Huang, C., Wang, Y., Li, X., Ren, L., Zhao, J., Hu, Y., Zhang, L., Fan, G., Xu, J., Gu, X., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(20)30183-5 · Cited by 48,884 · see citing works
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513 · see citing works
Wang, D., Hu, B., Hu, C., Zhu, F., Liu, X., Zhang, J., Wang, B., Xiang, H., Cheng, Z., Xiong, Y., et al. (2020) · JAMA · doi:10.1001/jama.2020.1585 · Cited by 20,148 · see citing works
Self-citationLa certeza de lo impredecible: Cultura Educación y Sociedad en tiempos de COVID19
Martínez González, M. (2020) · DSPACE System - Metalibrary (University of the Coast) · Cited by 18,463 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
Ai, T., Yang, Z., Hou, H., Zhan, C., Chen, C., Lv, W., Tao, Q., Sun, Z., & Xia, L. (2020) · Radiology · doi:10.1148/radiol.2020200642 · Cited by 5,697 · see citing works
Presumed Asymptomatic Carrier Transmission of COVID-19
Bai, Y., Yao, L., Wei, T., Tian, F., Jin, D., Chen, L., & Wang, M. (2020) · JAMA · doi:10.1001/jama.2020.2565 · Cited by 4,496 · see citing works
Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions
Agresti, A., & Coull, B. A. (1998) · The American Statistician · doi:10.1080/00031305.1998.10480550 · Cited by 3,223 · see citing works
MobileNetV2: Inverted Residuals and Linear BottlenecksPreprint
Sandler, M., Howard, A., Zhu, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.04381 · Cited by 2,255 · see citing works
Coronavirus Disease 2019 (COVID-19): A Perspective from China
Zu, Z. Y., Jiang, M. D., Xu, P. P., Chen, W., Ni, Q. Q., Lu, G. M., & Zhang, L. J. (2020) · Radiology · doi:10.1148/radiol.2020200490 · Cited by 2,070 · see citing works
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · see citing works
A deep learning algorithm using CT images to screen for Corona virus disease (COVID-19)
Wang, S., Kang, B., Ma, J., Zeng, X., Xiao, M., Guo, J., Cai, M., Yang, J., Li, Y., Meng, X., et al. (2021) · European Radiology · doi:10.1007/s00330-021-07715-1 · Cited by 1,668 · see citing works
Self-citationQiu, H., Wu, J., Hong, L., Luo, Y., Song, Q., & Dong, C. (2020) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(20)30198-5 · Cited by 1,282 · see citing works
The Role of Chest Imaging in Patient Management During the COVID-19 Pandemic
Rubin, G. D., Ryerson, C. J., Haramati, L. B., Sverzellati, N., Kanne, J. P., Raoof, S., Schluger, N. W., Volpi, A., Yim, J., Martin, I. B. K., et al. (2020) · CHEST Journal · doi:10.1016/j.chest.2020.04.003 · Cited by 1,249 · see citing works
Bai, H. X., Hsieh, B., Xiong, Z., Halsey, K., Choi, J. W., Tran, T. M. L., Pan, I., Shi, L.-B., Wang, D., Ji, M., et al. (2020) · Radiology · doi:10.1148/radiol.2020200823 · Cited by 1,232 · see citing works
Self-citationA Deep Learning System to Screen Novel Coronavirus Disease 2019 Pneumonia
Xu, X., Jiang, X., Ma, C., Du, P., Li, X., Lv, S., Yu, L., Ni, Q., Chen, Y., Su, J., et al. (2020) · Engineering · doi:10.1016/j.eng.2020.04.010 · Cited by 1,209 · see citing works
Self-citationRubin, G. D., Ryerson, C. J., Haramati, L. B., Sverzellati, N., Kanne, J. P., Raoof, S., Schluger, N. W., Volpi, A., Yim, J.-J., Martin, I. B. K., et al. (2020) · Radiology · doi:10.1148/radiol.2020201365 · Cited by 1,101 · see citing works
Li, Y., Yao, L., Li, J., Chen, L., Song, Y., Cai, Z., & Yang, C. (2020) · Journal of Medical Virology · doi:10.1002/jmv.25786 · Cited by 687 · see citing works
Self-citationChen, J., Wu, L., Zhang, J., Zhang, L., Gong, D., Zhao, Y., Chen, Q., Huang, S., Yang, M., Yang, X., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-76282-0 · Cited by 674 · see citing works
Chest CT Features of COVID-19 in Rome, Italy
Caruso, D., Zerunian, M., Polici, M., Pucciarelli, F., Polidori, T., Rucci, C., Guido, G., Bracci, B., Dominicis, C. D., & Laghi, A. (2020) · Radiology · doi:10.1148/radiol.2020201237 · Cited by 609 · see citing works
Deep learning for segmentation of brain tumors: Impact of cross‐institutional training and testing
AlBadawy, E. A., Saha, A., & Mazurowski, M. A. (2018) · Medical Physics · doi:10.1002/mp.12752 · Cited by 208 · see citing works
Shi, W., Peng, X., Liu, T., Cheng, Z., Lu, H., Yang, S., Zhang, J., Li, F., Wang, M., Zhang, X., et al. (2020) · Annals of Translational Medicine · doi:10.21037/atm-20-2464 · Cited by 82 · see citing works
Choi, H., Qi, X., Yoon, S. H., Park, S. J., Lee, K. H., Kim, J. Y., Lee, Y. K., Ko, H., Kim, K. H., Park, C. M., et al. (2020) · Radiology Cardiothoracic Imaging · doi:10.1148/ryct.2020204001 · Cited by 76 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [9]Li L, Qin L, Xu Z, Yin Y, Wang X, Kong B, et al. Artificial Intelligence Distinguishes COVID-19 from Community Acquired Pneumonia on Chest CT. Radiology. 0 (0): 200905. doi: 10.1148/radiol.2020200905. PubMed PMID: 32191588. · DOI · Paper page
- [24]Du Q. Clinical classification. Chinese Clinical Guidance for COVID-19 Pneumonia Diagnosis and Treatment (7th edition) [Internet]. 2020 April 6, 2020 April 8, 2020]. Available from: http://kjfy.meetingchina.org/msite/news/show/cn/3337.html.
Cited by
Sorted by citations
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00307-0 · Cited by 1,044 · see citing works
Yang, D., Xu, Z., Li, W., Myronenko, A., Roth, H. R., Harmon, S., Xu, S., Turkbey, B., Turkbey, E., Wang, X., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.101992 · Cited by 279 · see citing works
Shorfuzzaman, M., & Hossain, M. S. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2020.107700 · Cited by 217 · see citing works
Applications of artificial intelligence in battling against covid-19: A literature review
Tayarani N., M.-H. (2021) · Chaos, Solitons & Fractals · doi:10.1016/j.chaos.2020.110338 · Cited by 210 · see citing works
Eche, T., Schwartz, L. H., Mokrane, F.-Z., & Dercle, L. (2021) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2021210097 · Cited by 163 · see citing works
McGuire, J., De Cremer, D., & Van de Cruys, T. (2024) · Scientific Reports · doi:10.1038/s41598-024-69423-2 · Cited by 150 · see citing works
Must-have Qualities of Clinical Research on Artificial Intelligence and Machine Learning
Koçak, B., Cuocolo, R., Santos, D. P. D., Stanzione, A., & Ugga, L. (2023) · Balkan Medical Journal · doi:10.4274/balkanmedj.galenos.2022.2022-11-51 · Cited by 64 · see citing works
Galsgaard, A., Doorschodt, T., Holten, A.-L., Müller, F. C., Ploug Boesen, M., & Maas, M. (2022) · European Journal of Radiology · doi:10.1016/j.ejrad.2022.110231 · Cited by 38 · see citing works
- Authors
- Bai, H. X., Wang, R., Xiong, Z., Hsieh, B., Chang, K., Halsey, K., Tran, T. M. L., Choi, J. W., Wang, D.-C., Shi, L.-B., Mei, J., Jiang, X.-L., Pan, I., Zeng, Q.-H., Hu, P.-F., Li, Y.-H., Fu, F.-X., Huang, R. Y., Sebro, R., Yu, Q.-Z., Atalay, M. K., & Liao, W.-H.
- Affiliations
- Central South University (CN)
- Brigham and Women's Hospital (US)
- ORCID
- Venue
- Radiology
- Published
- 2020-09
- Volume
- 296
- Issue
- 3
- Pages
- E156-E165
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- PMC
- PMC7233483
- OpenAlex
- W3017451406
- Cited by
- 381
- References
- 23
- Citations by year
- 2026: 7 · 2025: 17 · 2024: 24 · 2023: 42 · 2022: 74 · 2021: 146 · 2020: 70
- FWCI
- 40.43
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Radiology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Cancer Institute (R03CA249554, F30CA239407)
- National Natural Science Foundation of China (81671676, 91959117)
- National Institute of Biomedical Imaging and Bioengineering (5T32EB1680)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- COVID-19 pandemic, non-COVID-19 pneumonia, chest CT, artificial intelligence, radiologist performance, deep learning, EfficientNet-B4, lung segmentation, diagnostic accuracy, AI-assisted diagnosis, inter-radiologist variability, model generalization, differential diagnosis of pneumonia, radiology residency
- MeSH terms
- Radiologists, Betacoronavirus, COVID-19, SARS-CoV-2, Adolescent, Adult, Aged, Aged, 80 and over, Artificial Intelligence, Child, Child, Preschool, China, Diagnosis, Differential, Female, Humans, Infant, Infant, Newborn, Lung, Male, Middle Aged, Pneumonia, Pneumonia, Viral, Retrospective Studies, Rhode Island, Sensitivity and Specificity, Radiography, Thoracic, Tomography, X-Ray Computed, Philadelphia, Coronavirus Infections, Young Adult, Pandemics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
