The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
DeepCOVID-XR: An Artificial Intelligence Algorithm to Detect COVID-19 on Chest Radiographs Trained and Tested on a Large U.S. Clinical Data Set
Open accessAbstract
Background There are characteristic findings of coronavirus disease 2019 (COVID-19) on chest images. An artificial intelligence (AI) algorithm to detect COVID-19 on chest radiographs might be useful for triage or infection control within a hospital setting, but prior reports have been limited by small data sets, poor data quality, or both. Purpose To present DeepCOVID-XR, a deep learning AI algorithm to detect COVID-19 on chest radiographs, that was trained and tested on a large clinical data set. Materials and Methods DeepCOVID-XR is an ensemble of convolutional neural networks developed to detect COVID-19 on frontal chest radiographs, with reverse-transcription polymerase chain reaction test results as the reference standard. The algorithm was trained and validated on 14 788 images (4253 positive for COVID-19) from sites across the Northwestern Memorial Health Care System from February 2020 to April 2020 and was then tested on 2214 images (1192 positive for COVID-19) from a single hold-out institution. Performance of the algorithm was compared with interpretations from five experienced thoracic radiologists on 300 random test images using the McNemar test for sensitivity and specificity and the DeLong test for the area under the receiver operating characteristic curve (AUC). Results A total of 5853 patients (mean age, 58 years ± 19 [standard deviation]; 3101 women) were evaluated across data sets. For the entire test set, accuracy of DeepCOVID-XR was 83%, with an AUC of 0.90. For 300 random test images (134 positive for COVID-19), accuracy of DeepCOVID-XR was 82%, compared with that of individual radiologists (range, 76%–81%) and the consensus of all five radiologists (81%). DeepCOVID-XR had a significantly higher sensitivity (71%) than one radiologist (60%, P < .001) and significantly higher specificity (92%) than two radiologists (75%, P < .001; 84%, P = .009). AUC of DeepCOVID-XR was 0.88 compared with the consensus AUC of 0.85 (P = .13 for comparison). With consensus interpretation as the reference standard, the AUC of DeepCOVID-XR was 0.95 (95% CI: 0.92, 0.98). Conclusion DeepCOVID-XR, an artificial intelligence algorithm, detected coronavirus disease 2019 on chest radiographs with a performance similar to that of experienced thoracic radiologists in consensus. © RSNA, 2020 Supplemental material is available for this article. See also the editorial by van Ginneken in this issue.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Literature has demonstrated how AI can revolutionize healthcare through its ability to perform various clinical tasks with a comparable or arguably even faster and with greater precision performance to that of humans [10–12].”
From Bias in artificial intelligence algorithms and recommendations for mitigation (2023) · cites this paper as [11]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,869
Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages
McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616
Wang, L., Lin, Z. Q., & Wong, A. M. (2020) · Scientific Reports · doi:10.1038/s41598-020-76550-z · Cited by 3,228
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Chest CT Findings in Coronavirus Disease-19 (COVID-19): Relationship to Duration of Infection
Bernheim, A., Mei, X., Huang, M., Yang, Y., Fayad, Z. A., Zhang, N., Diao, K., Lin, B., Zhu, X., Li, K., et al. (2020) · Radiology · doi:10.1148/radiol.2020200463 · Cited by 2,703
Li, L., Qin, L., Xu, Z., Yin, Y., Wang, X., Kong, B., Bai, J., Lu, Y., Fang, Z., Song, Q., et al. (2020) · Radiology · doi:10.1148/radiol.2020200905 · Cited by 1,927
Coronavirus Disease 2019 (COVID-19): A Systematic Review of Imaging Findings in 919 Patients
Salehi, S., Abedi, A., Balakrishnan, S., & Gholamrezanezhad, A. (2020) · American Journal of Roentgenology · doi:10.2214/ajr.20.23034 · Cited by 1,600
Frequency and Distribution of Chest Radiographic Findings in Patients Positive for COVID-19
Wong, H. Y. F., Lam, H. Y. S., Fong, A. H.-T., Leung, S. T., Chin, T. W.-Y., Lo, C. S.-Y., Lui, M. M., Lee, J. C. Y., Chiu, K. W.-H., Chung, T. W., et al. (2020) · Radiology · doi:10.1148/radiol.2020201160 · Cited by 1,524
Shi, F., Wang, J., Shi, J., Wu, Z., Wang, Q., Tang, Z., He, K., Shi, Y., & Shen, D. (2021) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.2987975 · Cited by 1,404
Bai, H. X., Hsieh, B., Xiong, Z., Halsey, K., Choi, J. W., Tran, T. M. L., Pan, I., Shi, L.-B., Wang, D., Ji, M., et al. (2020) · Radiology · doi:10.1148/radiol.2020200823 · Cited by 1,254
Imaging Profile of the COVID-19 Infection: Radiologic Findings and Literature Review
Ng, M., Lee, E. Y. P., Jin, Y., Yang, F., Li, X., Wang, H., Lui, M. M., Lo, C. S.-Y., Leung, B., Khong, P., et al. (2020) · Radiology Cardiothoracic Imaging · doi:10.1148/ryct.2020200034 · Cited by 1,163
Deep Learning COVID-19 Features on CXR Using Limited Training Data Sets
Oh, Y., Park, S. J., & Ye, J. C. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.2993291 · Cited by 912
COVID-19 Image Data CollectionPreprint
Cohen, J., Morrison, P., & Dao, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.11597 · Cited by 722
Interpreting a covid-19 test result
Watson, J., Whiting, P. F., & Brush, J. E. (2020) · BMJ · doi:10.1136/bmj.m1808 · Cited by 712
AI for radiographic COVID-19 detection selects shortcuts over signal
DeGrave, A. J., Janizek, J. D., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00338-7 · Cited by 631
Portable chest X-ray in coronavirus disease-19 (COVID-19): A pictorial review
Jacobi, A., Chung, M., Bernheim, A., & Eber, C. D. (2020) · Clinical Imaging · doi:10.1016/j.clinimag.2020.04.001 · Cited by 583
Serial Quantitative Chest CT Assessment of COVID-19: A Deep Learning Approach
Huang, L., Han, R., Ai, T., Yu, P., Kang, H. S., Tao, Q., & Xia, L. (2020) · Radiology Cardiothoracic Imaging · doi:10.1148/ryct.2020200075 · Cited by 429
The role of chest radiography in confirming covid-19 pneumonia
Cleverley, J., Piper, J., & Jones, M. M. (2020) · BMJ · doi:10.1136/bmj.m2426 · Cited by 233
COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System
Murphy, K., Smits, H., Knoops, A. J. G., Korst, M. B. J. M., Samson, T., Scholten, E. T., Schalekamp, S., Schaefer‐Prokop, C. M., Philipsen, R. H. H. M., Meijers, A., et al. (2020) · Radiology · doi:10.1148/radiol.2020201874 · Cited by 220
A role for CT in COVID-19? What data really tell us so far
Hope, M. D., Raptis, C. A., Shah, A. B., Hammer, M. M., & Henry, T. S. (2020) · The Lancet · doi:10.1016/s0140-6736(20)30728-5 · Cited by 188
Castiglioni, I., Ippolito, D., Interlenghi, M., Monti, C. B., Salvatore, C., Schiaffino, S., Polidori, A., Gandola, D. G., Messa, C., & Sardanelli, F. (2021) · European Radiology Experimental · doi:10.1186/s41747-020-00203-z · Cited by 145
Vancheri, S., Savietto, G., Ballati, F., Maggi, A., Canino, C., Bortolotto, C., Valentini, A., Dore, R., Stella, G. M., Corsico, A. G., et al. (2020) · European Radiology · doi:10.1007/s00330-020-06967-7 · Cited by 139
Generalizability of Deep Learning Tuberculosis Classifier to COVID-19 Chest Radiographs
Yi, P. H., Kim, T. K., & Lin, C. T. (2020) · Journal of Thoracic Imaging · doi:10.1097/rti.0000000000000532 · Cited by 40
Research works printed but not linked: 7
- Gardner L, Johns Hopkins Center for Systems Science and Engineering (CSSE) COVID-19 dashboard
- Weinstock MB, 2020, J Urgent Care Med, 14, 13
- 10.1136/bmj.m1328 · DOI · Paper page
- Shi F, 2020, IEEE Rev Biomed Eng
- American College of Radiology Web site
- Szegedy C, 2017, Proceedings of the thirty-first AAAI conference on artificial intelligence, 4278
- Watson J, 2020, BMJ, 369
Cited by
Sorted by citations
AI for radiographic COVID-19 detection selects shortcuts over signal
DeGrave, A. J., Janizek, J. D., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00338-7 · Cited by 631
Bias in artificial intelligence algorithms and recommendations for mitigation
Nazer, L. H., Zatarah, R., Waldrip, S., Ke, J. X. C., Moukheiber, M., Khanna, A. K., Hicklen, R. S., Moukheiber, L., Moukheiber, D., Ma, H., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000278 · Cited by 621
Deep learning for chest X-ray analysis: A survey
Çallı, E., Sogancioglu, E., van Ginneken, B., van Leeuwen, K. G., & Murphy, K. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102125 · Cited by 565
Robotics and artificial intelligence in healthcare during COVID-19 pandemic: a systematic review
Sarker, S., Jamal, L., Ahmed, S. F., & Irtisam, N. (2021) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2021.103902 · Cited by 199
A review of explainable and interpretable AI with applications in COVID‐19 imaging
Fuhrman, J. D., Gorre, N., Hu, Q., Li, H., Naqa, I. E., & Giger, M. L. (2021) · Medical Physics · doi:10.1002/mp.15359 · Cited by 124
Metadata
- Authors
- Wehbe, R. M., Sheng, J., Dutta, S., Chai, S., Dravid, A., Barutcu, S., Wu, Y., Cantrell, D. R., Xiao, N., Allen, B. D., MacNealy, G. A., Savas, H., Agrawal, R., Parekh, N., & Katsaggelos, A. K.
- Affiliations
- Northwestern University (US)
- Corresponding author
- Wehbe, Ramsey M.
- ORCID
- Venue
- Radiology
- Published
- 2021-04
- Volume
- 299
- Issue
- 1
- Pages
- E167-E176
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- PMC
- PMC7993244
- OpenAlex
- W3107538751
- Cited by
- 190
- References
- 30
- Citations by year
- 2026: 10 · 2025: 20 · 2024: 26 · 2023: 29 · 2022: 48 · 2021: 56
- FWCI
- 16.30
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Radiology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- COVID-19 detection, chest radiograph, convolutional neural networks, ensemble learning, artificial intelligence, radiologist interpretation, diagnostic accuracy, hospital triage, infection control, sensitivity and specificity comparison, COVID-19 pandemic
- MeSH terms
- COVID-19, SARS-CoV-2, Algorithms, Artificial Intelligence, Female, Humans, Lung, Male, Middle Aged, Radiographic Image Interpretation, Computer-Assisted, Sensitivity and Specificity, Radiography, Thoracic, United States, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
