DeepCOVID-XR: An artificial intelligence algorithm to detect COVID-19 on chest radiographs trained and tested on a large U.S. clinical data set
Open accessAbstract
Background There are characteristic findings of coronavirus disease 2019 (COVID-19) on chest images. An artificial intelligence (AI) algorithm to detect COVID-19 on chest radiographs might be useful for triage or infection control within a hospital setting, but prior reports have been limited by small data sets, poor data quality, or both. Purpose To present DeepCOVID-XR, a deep learning AI algorithm to detect COVID-19 on chest radiographs, that was trained and tested on a large clinical data set. Materials and Methods DeepCOVID-XR is an ensemble of convolutional neural networks developed to detect COVID-19 on frontal chest radiographs, with reverse-transcription polymerase chain reaction test results as the reference standard. The algorithm was trained and validated on 14 788 images (4253 positive for COVID-19) from sites across the Northwestern Memorial Health Care System from February 2020 to April 2020 and was then tested on 2214 images (1192 positive for COVID-19) from a single hold-out institution. Performance of the algorithm was compared with interpretations from five experienced thoracic radiologists on 300 random test images using the McNemar test for sensitivity and specificity and the DeLong test for the area under the receiver operating characteristic curve (AUC). Results A total of 5853 patients (mean age, 58 years ± 19 [standard deviation]; 3101 women) were evaluated across data sets. For the entire test set, accuracy of DeepCOVID-XR was 83%, with an AUC of 0.90. For 300 random test images (134 positive for COVID-19), accuracy of DeepCOVID-XR was 82%, compared with that of individual radiologists (range, 76%–81%) and the consensus of all five radiologists (81%). DeepCOVID-XR had a significantly higher sensitivity (71%) than one radiologist (60%, P < .001) and significantly higher specificity (92%) than two radiologists (75%, P < .001; 84%, P = .009). AUC of DeepCOVID-XR was 0.88 compared with the consensus AUC of 0.85 (P = .13 for comparison). With consensus interpretation as the reference standard, the AUC of DeepCOVID-XR was 0.95 (95% CI: 0.92, 0.98). Conclusion DeepCOVID-XR, an artificial intelligence algorithm, detected coronavirus disease 2019 on chest radiographs with a performance similar to that of experienced thoracic radiologists in consensus. © RSNA, 2020 Supplemental material is available for this article. See also the editorial by van Ginneken in this issue.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI can perform various clinical tasks with comparable or faster and greater precision performance to humans.
“Literature has demonstrated how AI can revolutionize healthcare through its ability to perform various clinical tasks with a comparable or arguably even faster and with greater precision performance to that of humans [10–12].”
From Bias in artificial intelligence algorithms and recommendations for mitigation (2023) · cites this paper as [11]
The Results (Comparison with Expert Thoracic Radiologists) report similar performance and faster analysis for DeepCOVID-XR on one task: detecting COVID-19 on chest radiographs. They do not establish greater precision than the radiologists or performance across various clinical tasks. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513 · see citing works
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · see citing works
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · see citing works
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,869 · see citing works
Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages
McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754 · see citing works
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · see citing works
Wang, L., Lin, Z. Q., & Wong, A. M. (2020) · Scientific Reports · doi:10.1038/s41598-020-76550-z · Cited by 3,151 · see citing works
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
Chest CT Findings in Coronavirus Disease-19 (COVID-19): Relationship to Duration of Infection
Bernheim, A., Mei, X., Huang, M., Yang, Y., Fayad, Z. A., Zhang, N., Diao, K., Lin, B., Zhu, X., Li, K., et al. (2020) · Radiology · doi:10.1148/radiol.2020200463 · Cited by 2,658 · see citing works
Li, L., Qin, L., Xu, Z., Yin, Y., Wang, X., Kong, B., Bai, J., Lu, Y., Fang, Z., Song, Q., et al. (2020) · Radiology · doi:10.1148/radiol.2020200905 · Cited by 1,927 · see citing works
Coronavirus Disease 2019 (COVID-19): A Systematic Review of Imaging Findings in 919 Patients
Salehi, S., Abedi, A., Balakrishnan, S., & Gholamrezanezhad, A. (2020) · American Journal of Roentgenology · doi:10.2214/ajr.20.23034 · Cited by 1,600 · see citing works
Frequency and Distribution of Chest Radiographic Findings in Patients Positive for COVID-19
Wong, H. Y. F., Lam, H. Y. S., Fong, A. H.-T., Leung, S. T., Chin, T. W.-Y., Lo, C. S.-Y., Lui, M. M., Lee, J. C. Y., Chiu, K. W.-H., Chung, T. W., et al. (2020) · Radiology · doi:10.1148/radiol.2020201160 · Cited by 1,485 · see citing works
Shi, F., Wang, J., Shi, J., Wu, Z., Wang, Q., Tang, Z., He, K., Shi, Y., & Shen, D. (2021) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.2987975 · Cited by 1,404 · see citing works
Bai, H. X., Hsieh, B., Xiong, Z., Halsey, K., Choi, J. W., Tran, T. M. L., Pan, I., Shi, L.-B., Wang, D., Ji, M., et al. (2020) · Radiology · doi:10.1148/radiol.2020200823 · Cited by 1,232 · see citing works
Imaging Profile of the COVID-19 Infection: Radiologic Findings and Literature Review
Ng, M., Lee, E. Y. P., Jin, Y., Yang, F., Li, X., Wang, H., Lui, M. M., Lo, C. S.-Y., Leung, B., Khong, P., et al. (2020) · Radiology Cardiothoracic Imaging · doi:10.1148/ryct.2020200034 · Cited by 1,141 · see citing works
Deep Learning COVID-19 Features on CXR Using Limited Training Data Sets
Oh, Y., Park, S. J., & Ye, J. C. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.2993291 · Cited by 897 · see citing works
COVID-19 Image Data CollectionPreprint
Cohen, J., Morrison, P., & Dao, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.11597 · Cited by 722 · see citing works
Interpreting a covid-19 test result
Watson, J., Whiting, P. F., & Brush, J. E. (2020) · BMJ · doi:10.1136/bmj.m1808 · Cited by 712 · see citing works
AI for radiographic COVID-19 detection selects shortcuts over signal
DeGrave, A. J., Janizek, J. D., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00338-7 · Cited by 631 · see citing works
Portable chest X-ray in coronavirus disease-19 (COVID-19): A pictorial review
Jacobi, A., Chung, M., Bernheim, A., & Eber, C. D. (2020) · Clinical Imaging · doi:10.1016/j.clinimag.2020.04.001 · Cited by 583 · see citing works
Serial Quantitative Chest CT Assessment of COVID-19: A Deep Learning Approach
Huang, L., Han, R., Ai, T., Yu, P., Kang, H. S., Tao, Q., & Xia, L. (2020) · Radiology Cardiothoracic Imaging · doi:10.1148/ryct.2020200075 · Cited by 424 · see citing works
COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System
Murphy, K., Smits, H., Knoops, A. J. G., Korst, M. B. J. M., Samson, T., Scholten, E. T., Schalekamp, S., Schaefer‐Prokop, C. M., Philipsen, R. H. H. M., Meijers, A., et al. (2020) · Radiology · doi:10.1148/radiol.2020201874 · Cited by 218 · see citing works
The role of chest radiography in confirming covid-19 pneumonia
Cleverley, J., Piper, J., & Jones, M. M. (2020) · BMJ · doi:10.1136/bmj.m2426 · Cited by 218 · see citing works
A role for CT in COVID-19? What data really tell us so far
Hope, M. D., Raptis, C. A., Shah, A. B., Hammer, M. M., & Henry, T. S. (2020) · The Lancet · doi:10.1016/s0140-6736(20)30728-5 · Cited by 188 · see citing works
Castiglioni, I., Ippolito, D., Interlenghi, M., Monti, C. B., Salvatore, C., Schiaffino, S., Polidori, A., Gandola, D. G., Messa, C., & Sardanelli, F. (2021) · European Radiology Experimental · doi:10.1186/s41747-020-00203-z · Cited by 145 · see citing works
Vancheri, S., Savietto, G., Ballati, F., Maggi, A., Canino, C., Bortolotto, C., Valentini, A., Dore, R., Stella, G. M., Corsico, A. G., et al. (2020) · European Radiology · doi:10.1007/s00330-020-06967-7 · Cited by 139 · see citing works
Generalizability of Deep Learning Tuberculosis Classifier to COVID-19 Chest Radiographs
Yi, P. H., Kim, T. K., & Lin, C. T. (2020) · Journal of Thoracic Imaging · doi:10.1097/rti.0000000000000532 · Cited by 42 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [2]Weinstock MB, Echenique A, Russell JW, et al. Chest x-ray findings in 636 ambulatory patients with COVID-19 presenting to an urgent care center: a normal chest x-ray is no guarantee. J Urgent Care Med 2020; 14 (7): 13 – 18.
- [20]Szegedy C, Ioffe S, Vanhoucke V, Alemi AA. Inception-v4, inception-ResNet and the impact of residual connections on learning. In: Proceedings of the thirty-first AAAI conference on artificial intelligence. Menlo Park, CA: Association for the Advancement of Artificial Intelligence, 2017; 4278 – 4284.
Cited by
Sorted by citations
AI for radiographic COVID-19 detection selects shortcuts over signal
DeGrave, A. J., Janizek, J. D., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00338-7 · Cited by 631 · see citing works
Bias in artificial intelligence algorithms and recommendations for mitigation
Nazer, L. H., Zatarah, R., Waldrip, S., Ke, J. X. C., Moukheiber, M., Khanna, A. K., Hicklen, R. S., Moukheiber, L., Moukheiber, D., Ma, H., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000278 · Cited by 621 · see citing works
Deep learning for chest X-ray analysis: A survey
Çallı, E., Sogancioglu, E., van Ginneken, B., van Leeuwen, K. G., & Murphy, K. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102125 · Cited by 531 · see citing works
Robotics and artificial intelligence in healthcare during COVID-19 pandemic: a systematic review
Sarker, S., Jamal, L., Ahmed, S. F., & Irtisam, N. (2021) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2021.103902 · Cited by 199 · see citing works
A review of explainable and interpretable AI with applications in COVID‐19 imaging
Fuhrman, J. D., Gorre, N., Hu, Q., Li, H., Naqa, I. E., & Giger, M. L. (2021) · Medical Physics · doi:10.1002/mp.15359 · Cited by 120 · see citing works
Ejaz, H., McGrath, H., Wong, B. L., Guise, A., Vercauteren, T., & Shapey, J. (2022) · DIGITAL HEALTH · doi:10.1177/20552076221089099 · Cited by 105 · see citing works
Nillmani, Sharma, N., Saba, L., Khanna, N. N., Kalra, M. K., Fouda, M. M., & Suri, J. S. (2022) · Diagnostics · doi:10.3390/diagnostics12092132 · Cited by 101 · see citing works
Digital imaging, technologies and artificial intelligence applications during COVID-19 pandemic
Alhasan, M., & Hasaneen, M. (2021) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2021.101933 · Cited by 89 · see citing works
Artificial Intelligence‐Based Medical Sensors for Healthcare System
Chen, M., Cui, D., Haick, H., & Tang, N. (2023) · Advanced Sensor Research · doi:10.1002/adsr.202300009 · Cited by 85 · see citing works
Explainable AI, But Explainable to Whom? An Exploratory Case Study of xAI in Healthcare
Gerlings, J., Jensen, M. S., & Shollo, A. (2021) · Intelligent systems reference library · doi:10.1007/978-3-030-83620-7_7 · Cited by 85 · see citing works
Thoracic imaging tests for the diagnosis of COVID-19
Ebrahimzadeh, S., Islam, N., Dawit, H., Salameh, J., Kazi, S., Fabiano, N., Treanor, L. M., Absi, M., Ahmad, F., Rooprai, P., et al. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013639.pub5 · Cited by 79 · see citing works
Dong, J., Wu, H., Zhou, D., Li, K., Zhang, Y., Ji, H., Tong, Z., Lou, S., & Liu, Z. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01757-0 · Cited by 74 · see citing works
Wang, W., Zhang, X., Wang, S., & Zhang, Y. (2022) · Systems Science & Control Engineering · doi:10.1080/21642583.2022.2045645 · Cited by 66 · see citing works
Artificial Intelligence-Based Detection of Pneumonia in Chest Radiographs
Becker, J., Decker, J. A., Römmele, C., Kahn, M., Messmann, H., Wehler, M., Schwarz, F., Kroencke, T. J., & Scheurig‐Muenkler, C. (2022) · Diagnostics · doi:10.3390/diagnostics12061465 · Cited by 68 · see citing works
Explainable Vision Transformers and Radiomics for COVID-19 Detection in Chest X-rays
Chetoui, M., & Akhloufi, M. A. (2022) · Journal of Clinical Medicine · doi:10.3390/jcm11113013 · Cited by 60 · see citing works
Innovative applications of artificial intelligence during the COVID-19 pandemic
Lv, C., Guo, W., Yin, X., Liu, L., Huang, X., Li, S., & Zhang, L. (2024) · Infectious Medicine · doi:10.1016/j.imj.2024.100095 · Cited by 59 · see citing works
A deep-learning model for identifying fresh vertebral compression fractures on digital radiography
Chen, W., Liu, X., Li, K., Luo, Y., Bai, S., Wu, J., Chen, W., Dong, M., & Guo, D. (2021) · European Radiology · doi:10.1007/s00330-021-08247-4 · Cited by 56 · see citing works
Ullah, N., Khan, J. A., Almakdi, S., Khan, M. S., Alshehri, M., Alboaneen, D., & Raza, A. (2022) · Applied Sciences · doi:10.3390/app12126269 · Cited by 50 · see citing works
Hwang, E. J., Kim, K. B., Kim, J. Y., Lim, J., Nam, J. G., Choi, H., Kim, H., Yoon, S. H., Goo, J. M., & Park, C. M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252440 · Cited by 48 · see citing works
Yang, H., Wang, L., Xu, Y., & Liu, X. (2022) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01676-7 · Cited by 45 · see citing works
Medical imaging and computational image analysis in COVID-19 diagnosis: A review
Nabavi, S., Ejmalian, A., Moghaddam, M. E., Abin, A. A., Frangi, A. F., Mohammadi, M. H., & Rad, H. S. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104605 · Cited by 43 · see citing works
Quality gaps in public pancreas imaging datasets: Implications & challenges for AI applications
Suman, G., Patra, A., Korfiatis, P., Majumder, S., Chari, S. T., Truty, M. J., Fletcher, J. G., & Goenka, A. H. (2021) · Pancreatology · doi:10.1016/j.pan.2021.03.016 · Cited by 41 · see citing works
A deep learning segmentation-classification pipeline for X-ray-based COVID-19 diagnosis
Hertel, R., & Benlamri, R. (2022) · Biomedical Engineering Advances · doi:10.1016/j.bea.2022.100041 · Cited by 36 · see citing works
A Novel Multi-Stage Residual Feature Fusion Network for Detection of COVID-19 in Chest X-Ray Images
Fang, Z., Ren, J., MacLellan, C., Li, H., Zhao, H., Hussain, A., & Fortino, G. (2021) · IEEE Transactions on Molecular Biological and Multi-Scale Communications · doi:10.1109/tmbmc.2021.3099367 · Cited by 36 · see citing works
Chetoui, M., Akhloufi, M. A., Yousefi, B., & Bouattane, E. M. (2021) · Big Data and Cognitive Computing · doi:10.3390/bdcc5040073 · Cited by 34 · see citing works
A comprehensive review of imaging findings in COVID-19 - status in early 2021
Afshar‐Oromieh, A., Prosch, H., Schaefer‐Prokop, C. M., Bohn, K. P., Alberts, I., Mingels, C., Thurnher, M. M., Cumming, P. D., Shi, K., Peters, A. A., et al. (2021) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-021-05375-3 · Cited by 34 · see citing works
Khan, I. U., Aslam, N., Anwar, T., Alsaif, H. S., Chrouf, S. M. B., Alzahrani, N. A., Alamoudi, F. A., Kamaleldin, M. M. A., & Awary, K. B. (2022) · Sensors · doi:10.3390/s22020669 · Cited by 33 · see citing works
Niehoff, J. H., Kalaitzidis, J., Kröger, J. R., Schoenbeck, D., Borggrefe, J., & Michael, A. E. (2023) · Scientific Reports · doi:10.1038/s41598-023-30521-2 · Cited by 31 · see citing works
Nishio, M., Kobayashi, D., Nishioka, E., Matsuo, H., Urase, Y., Onoue, K., Ishikura, R., Kitamura, Y., Sakai, E., Tomita, M., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-11990-3 · Cited by 35 · see citing works
Applications and challenges of AI-based algorithms in the COVID-19 pandemic
Khemasuwan, D., & Colt, H. G. (2021) · BMJ Innovations · doi:10.1136/bmjinnov-2020-000648 · Cited by 31 · see citing works
Rethinking Densely Connected Convolutional Networks for Diagnosing Infectious Diseases
Podder, P., Alam, F. B., Mondal, M. R. H., Hasan, M. J., Rohan, A., & Bharati, S. (2023) · Computers · doi:10.3390/computers12050095 · Cited by 30 · see citing works
Lessons learned in transitioning to AI in the medical imaging of COVID-19
Naqa, I. M. E., Hu, Q., Chen, W., Li, H., Fuhrman, J. D., Gorre, N., & Giger, M. L. (2021) · Journal of medical imaging · doi:10.1117/1.jmi.8.s1.010902 · Cited by 29 · see citing works
Li, F., Lu, X., & Yuan, J. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3134270 · Cited by 28 · see citing works
Khanna, N. N., Maindarkar, M. A., Puvvula, A., Paul, S., Bhagawati, M., Ahluwalia, P., Ruzsa, Z., Sharma, A., Munjral, S., Kolluri, R., et al. (2022) · Journal of Cardiovascular Development and Disease · doi:10.3390/jcdd9080268 · Cited by 28 · see citing works
Puniya, B. L. (2025) · Journal of Molecular Biology · doi:10.1016/j.jmb.2025.169181 · Cited by 29 · see citing works
Future Directions in Artificial Intelligence
Saboury, B., Morris, M. A., & Siegel, E. L. (2021) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2021.07.008 · Cited by 27 · see citing works
COVID-19 detection on chest radiographs using feature fusion based deep learning
Bayram, F., & Eleyan, A. (2022) · Signal Image and Video Processing · doi:10.1007/s11760-021-02098-8 · Cited by 28 · see citing works
Sun, J., Shi, W., Giuste, F., Vaghani, Y. S., Tang, L., & Wang, M. D. (2023) · Scientific Reports · doi:10.1038/s41598-023-46493-2 · Cited by 27 · see citing works
Generalizable disease detection using model ensemble on chest X-ray images
Abad, M., Casas-Roma, J., & Prados, F. (2024) · Scientific Reports · doi:10.1038/s41598-024-56171-6 · Cited by 26 · see citing works
Awassa, L., Jdey, I., Dhahri, H., Hcini, G., Mahmood, A., Othman, E. M. A., & Haneef, M. (2022) · Sensors · doi:10.3390/s22051890 · Cited by 25 · see citing works
Tracking the pipeline: immunoinformatics and the COVID-19 vaccine design
Rezaei, S., Sefidbakht, Y., & Uskoković, V. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab241 · Cited by 24 · see citing works
Kriza, C., Amenta, V., Zenié, A., Panidis, D., Chassaigne, H., Urbán, P., Holzwarth, U., Sauer, A. V., Reina, V., & Griesinger, C. B. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.110028 · Cited by 24 · see citing works
Wu, Y., Rocha, B. M., Kaimakamis, E., Cheimariotis, G.-A., Petmezas, G., Chatzis, E., Kilintzis, V., Stefanopoulos, L., Pessoa, D., Marques, A., et al. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121089 · Cited by 23 · see citing works
Self-citationSalem, H. S., Mead, M. A., & Eltaweel, G. (2024) · Journal of Computer and Communications · doi:10.4236/jcc.2024.123010 · Cited by 25 · see citing works
Detection of COVID-19 Using X-ray Image Classification
Garlapati, K., Kota, N., Mondreti, Y. S., Gutha, P., & Nair, A. K. (2021) · International Conference on Trends in Electronics and Informatics (ICOEI) · doi:10.1109/icoei51242.2021.9452745 · Cited by 22 · see citing works
COV-SNET: A deep learning model for X-ray-based COVID-19 classification
Hertel, R., & Benlamri, R. (2021) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2021.100620 · Cited by 21 · see citing works
JUWELS Booster – A Supercomputer for Large-Scale AI Research
Kesselheim, S., Herten, A., Krajsek, K., Ebert, J., Jitsev, J., Cherti, M., Langguth, M., Gong, B., Stadtler, S., Mozaffari, A., et al. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-90539-2_31 · Cited by 21 · see citing works
Predicting clinical outcomes in COVID-19 using radiomics on chest radiographs
Varghese, B., Shin, H., Desai, B., Gholamrezanezhad, A., Lei, X., Perkins, M., Oberai, A., Nanda, N., Cen, S., & Duddalwar, V. (2021) · British Journal of Radiology · doi:10.1259/bjr.20210221 · Cited by 23 · see citing works
medXGAN: Visual Explanations for Medical Classifiers through a Generative Latent Space
Dravid, A. V., Schiffers, F., Gong, B., & Katsaggelos, A. K. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00331 · Cited by 21 · see citing works
Self-citationHaynes, S. C., Johnston, P., & Elyan, E. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18543-y · Cited by 23 · see citing works
Abdel‐Basset, M., Hawash, H., Moustafa, N., & Elkomy, O. M. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.10.027 · Cited by 20 · see citing works
Kim, N. J., Oh, J., Kim, S., Cha, J., Choi, J., Im, S. G., Choi, S., & Jang, B. C. (2024) · Advanced Functional Materials · doi:10.1002/adfm.202305136 · Cited by 20 · see citing works
Hu, Q., Drukker, K., & Giger, M. L. (2021) · Journal of medical imaging · doi:10.1117/1.jmi.8.s1.014503 · Cited by 17 · see citing works
Potential diagnosis of COVID-19 from chest X-ray and CT findings using semi-supervised learning
Sahoo, P., Roy, I., Ahlawat, R., Irtiza, S., & Khan, L. R. (2021) · Physical and Engineering Sciences in Medicine · doi:10.1007/s13246-021-01075-2 · Cited by 18 · see citing works
MultiCOVID: a multi modal deep learning approach for COVID-19 diagnosis
Hardy-Werbin, M., Maiques, J. M., Busto, M., Cirera, I., Aguirre, A., García-Gisbert, N., Zuccarino, F., Carbullanca, S., Bellido, L. A. D. C., Ramal, D., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-46126-8 · Cited by 18 · see citing works
van Ginneken, B. (2020) · Radiology · doi:10.1148/radiol.2020204238 · Cited by 17 · see citing works
Ippolito, D., Maino, C., Gandola, D. G., Franco, P. N., Miron, R. F., Barbu, V., Bologna, M., Corso, R., & Breaban, M. E. (2023) · Diseases · doi:10.3390/diseases11040171 · Cited by 15 · see citing works
Transition toward Healthcare 5.0: Impact of COVID-19 in the healthcare industry
Raisa, J. F., Rahman, M. S., Mahmud, I., Kaiser, M. S., & Han, D. S. (2025) · ICT Express · doi:10.1016/j.icte.2025.04.002 · Cited by 17 · see citing works
Laddha, S., Mnasri, S., Alghamdi, M., Kumar, V., Kaur, M., Alrashidi, M., Almuhaimeed, A., Alshehri, A., Alrowaily, M. A., & Alkhazi, I. S. (2022) · Diagnostics · doi:10.3390/diagnostics12081880 · Cited by 15 · see citing works
Sim, J. Z. T., Ting, Y. H., Tang, Y., Feng, Y., Lei, X., Wang, X., Chen, W., Huang, S., Wong, S. T., Lu, Z., et al. (2022) · Healthcare · doi:10.3390/healthcare10010175 · Cited by 14 · see citing works
L’intelligence artificielle peut-elle être une innovation responsable ?
Vuarin, L., Lopes, P. G., & Massé, D. (2023) · Innovations · doi:10.3917/inno.pr2.0153 · Cited by 11 · see citing works
Aasem, M., & Iqbal, M. J. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1366415 · Cited by 23 · see citing works
Giansanti, D., Rossi, I., & Monoscalco, L. (2021) · Healthcare · doi:10.3390/healthcare9030331 · Cited by 13 · see citing works
Cushnan, D., Berka, R., Bertolli, O., Williams, P., Schofield, D., Joshi, I., Favaro, A., Halling‐Brown, M., Imreh, G., Jefferson, E., et al. (2021) · Digital Health · doi:10.1177/20552076211048654 · Cited by 13 · see citing works
Ben Ahmed, K., Hall, L. O., Goldgof, D. B., & Fogarty, R. (2022) · IEEE Access · doi:10.1109/access.2022.3193700 · Cited by 12 · see citing works
Artificial Intelligence: A Next-Level Approach in Confronting the COVID-19 Pandemic
Mahalakshmi, V. J., Balobaid, A. S., Kanisha, B., Sasirekha, R., & Raja, M. R. (2023) · Healthcare · doi:10.3390/healthcare11060854 · Cited by 12 · see citing works
Yousefpanah, K., Ebadi, M. J., Sabzekar, S., Zakaria, N. H., Osman, N. A., & Ahmadian, A. (2024) · Acta Tropica · doi:10.1016/j.actatropica.2024.107277 · Cited by 12 · see citing works
Maqbool, T., Ishaq, H., Shakeel, S., Nisa, A. Z. U., Rehman, H., Kashif, S., Sadia, H., Naveed, S., Mumtaz, N., Siddiqui, S., et al. (2025) · PLoS ONE · doi:10.1371/journal.pone.0314045 · Cited by 19 · see citing works
Leveraging federated learning and edge computing for pandemic-resilient healthcare
Choudhury, A., Sarma, K. K., Gulvanskii, V. V., Kaplun, D., & Dutta, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-00199-9 · Cited by 15 · see citing works
Design and Optimization of Personalized Education System Based on Intelligent Algorithms
Zhu, J., Liao, Y., & Lu, D. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.063 · Cited by 11 · see citing works
Imaging in the COVID-19 era: Lessons learned during a pandemic
Sideris, G. A., Nikolakea, M., Karanikola, A.-E., Konstantinopoulou, S., Giannis, D., & Modahl, L. E. (2021) · World Journal of Radiology · doi:10.4329/wjr.v13.i6.192 · Cited by 10 · see citing works
CovXr: Automated Detection of COVID-19 Pneumonia in Chest X-Rays through Machine Learning
Shenoy, V., & Malik, S. B. (2021) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci50451.2021.9659919 · Cited by 11 · see citing works
Drozdov, I. A., Szubert, B., Reda, E., Makary, P., Forbes, D., Chang, S. L., Ezhil, A., Puttagunta, S. A., Hall, M., Carlin, C. M., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-99986-3 · Cited by 11 · see citing works
Kuo, M., Chiu, K. W.-H., Wang, D. S., Larici, A. R., Poplavskiy, D., Valentini, A., Napoli, A., Borghesi, A., Ligabue, G., Fang, B., et al. (2022) · European Radiology · doi:10.1007/s00330-022-08969-z · Cited by 11 · see citing works
Artificial Intelligence Assisted Drug Research and Development
Soni, K., & Hasija, Y. (2022) · IEEE Delhi Section Conference (DELCON) · doi:10.1109/delcon54057.2022.9753179 · Cited by 11 · see citing works
Deep Learning Models to Predict Fatal Pneumonia Using Chest X-Ray Images
Anai, S., Hisasue, J., Takaki, Y., & Hara, N. (2022) · Canadian Respiratory Journal · doi:10.1155/2022/8026580 · Cited by 11 · see citing works
Transparent and trustworthy interpretation of COVID-19 features in chest X-rays using explainable AI
Kinger, S., & Kulkarni, V. Y. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19755-y · Cited by 10 · see citing works
Radočaj, P., & Martinović, G. (2025) · Electronics · doi:10.3390/electronics14091899 · Cited by 11 · see citing works
Covid-19 Detection by Wavelet Entropy and Jaya
Wang, W. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-84532-2_45 · Cited by 10 · see citing works
Miyazaki, A., Ikejima, K., Nishio, M., Yabuta, M., Matsuo, H., Onoue, K., Matsunaga, T., Nishioka, E., Kono, A. K., Yamada, D., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-44818-9 · Cited by 10 · see citing works
Zhang, Q., Chen, Z., Liu, G., Zhang, W., Du, Q., Tan, J., & Gao, Q. (2021) · Infection and Drug Resistance · doi:10.2147/idr.s296346 · Cited by 11 · see citing works
Li, M., Chang, K., Mei, X., Bernheim, A., Chung, M., Steinberger, S., Kalpathy–Cramer, J., & Little, B. P. (2021) · American Journal of Roentgenology · doi:10.2214/ajr.21.26717 · Cited by 10 · see citing works
Deep Learning Applied to Chest Radiograph Classification—A COVID-19 Pneumonia Experience
Furtado, A. N., Andrade, L., Frı́as, D., Maia, T., Badaró, R. J. D. S., & Nascimento, E. G. S. (2022) · Applied Sciences · doi:10.3390/app12083712 · Cited by 10 · see citing works
Yoo, S.-J., Kim, H., Witanto, J. N., Inui, S., Yoon, J.-H., Lee, K.-D., Choi, Y. W., Goo, J. M., & Yoon, S. H. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.110858 · Cited by 9 · see citing works
Neuromorphic chips for biomedical engineering
Wang, K., Ren, S., Jia, Y., Yan, X., Wang, L., & Fan, Y. (2025) · Mechanobiology in Medicine · doi:10.1016/j.mbm.2025.100133 · Cited by 9 · see citing works
Catalano, M., Bortolotto, C., Nicora, G., Achilli, M. F., Consonni, A., Ruongo, L., Callea, G., Tito, A. L., Biasibetti, C., Donatelli, A., et al. (2023) · European Journal of Radiology Open · doi:10.1016/j.ejro.2023.100497 · Cited by 10 · see citing works
Wu, Y., Iorga, M., Badhe, S., Zhang, J., Cantrell, D. R., Tanhehco, E. J., Szrama, N. P., Naidech, A. M., Drakopoulos, M., Hasan, S. T., et al. (2024) · Radiology Artificial Intelligence · doi:10.1148/ryai.230296 · Cited by 9 · see citing works
Self-citationCovid-19 Diagnosis using Deep Learning Approaches: A Systematic Review
Abdullah, T. H., & Abdullah, B. H. (2024) · Mesopotamian Journal of Artificial Intelligence in Healthcare · doi:10.58496/mjaih/2024/012 · Cited by 8 · see citing works
Kilintzis, V., Beredimas, N., Kaimakamis, E., Stefanopoulos, L., Chatzis, E., Jahaj, E., Bitzani, M., Kotanidou, Α., Katsaggelos, A. K., & Maglaveras, N. (2022) · Healthcare · doi:10.3390/healthcare10020276 · Cited by 8 · see citing works
Self-citationDeep Efficient Neural Networks for Explainable COVID-19 Detection on CXR Images
Chetoui, M., & Akhloufi, M. A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-79457-6_29 · Cited by 8 · see citing works
A 178-clinical-center experiment of integrating AI solutions for lung pathology diagnosis
Ibragimov, B., Arzamasov, K. M., Maksudov, B., Kiselev, S., Mongolin, A., Mustafaev, T., Ibragimova, D., Evteeva, K., Андрейченко, А. Е., & Morozov, S. P. (2023) · Scientific Reports · doi:10.1038/s41598-023-27397-7 · Cited by 7 · see citing works
Potential of artificial intelligence to accelerate diagnosis and drug discovery for COVID-19
Mikkili, I., Peele, K. A., Venkateswarulu, T. C., Kodali, V. P., Macamdas, D. S. S., & Sreerama, K. (2021) · PeerJ · doi:10.7717/peerj.12073 · Cited by 8 · see citing works
Wang, T., Chen, Z., Shang, Q., Ma, C., Chen, X., & Xiao, E. (2021) · Diagnostics · doi:10.3390/diagnostics11101924 · Cited by 7 · see citing works
Indirect supervision applied to COVID-19 and pneumonia classification
Данилов, В., Proutski, A., Karpovsky, A., Kirpich, A., Litmanovich, D. E., Nefaridze, D., Talalov, O., Semyonov, S., Koniukhovskii, V., Shvartc, V., et al. (2021) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2021.100835 · Cited by 7 · see citing works
A comparison of Covid-19 early detection between convolutional neural networks and radiologists
Albiol, A., Albiol, F., Paredes, R., Plasencia-Martínez, J. M., Barrio, A. B., Santos, J. M. G., Tortajada, S., Montaño, V. M. G., Godoy, C. R., Gómez, S. F., et al. (2022) · Insights into Imaging · doi:10.1186/s13244-022-01250-3 · Cited by 7 · see citing works
Geroski, T., Ranković, V., Pavić, O., Dašić, L., Petrovic, M. D., Milovanovic, D. R., & Filipović, N. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107103 · Cited by 6 · see citing works
Explainable AI and Its Applications in Healthcare
Sarkar, A. (2022) · Intelligent systems reference library · doi:10.1007/978-3-031-12807-3_6 · Cited by 6 · see citing works
Abdulah, H., Huber, B., Abdallah, H., Palese, L. L., Hamid Soltanian-zadeh, A., & Gatti, D. L. (2022) · medRxiv · doi:10.1101/2022.03.13.22272311 · Cited by 6 · see citing works
Lee, L., Lin, C. S., Hsu, C.-J., Lin, H.-H., Lin, T.-C., Liu, Y.-H., & Hu, J. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01309-1 · Cited by 5 · see citing works
- Authors
- Wehbe, R. M., Sheng, J., Dutta, S., Chai, S., Dravid, A., Barutcu, S., Wu, Y., Cantrell, D. R., Xiao, N., Allen, B. D., MacNealy, G. A., Savas, H., Agrawal, R., Parekh, N., & Katsaggelos, A. K.
- Affiliations
- Northwestern University (US)
- Corresponding author
- Wehbe, Ramsey M.
- ORCID
- Venue
- Radiology
- Published
- 2021-04
- Volume
- 299
- Issue
- 1
- Pages
- E167-E176
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- PMC
- PMC7993244
- OpenAlex
- W3107538751
- Cited by
- 190
- References
- 30
- Citations by year
- 2026: 10 · 2025: 20 · 2024: 26 · 2023: 29 · 2022: 48 · 2021: 56 · 2020: 1
- FWCI
- 16.30
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Radiology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- COVID-19 detection, chest radiograph, convolutional neural networks, ensemble learning, artificial intelligence, radiologist interpretation, diagnostic accuracy, hospital triage, infection control, sensitivity and specificity comparison, COVID-19 pandemic
- MeSH terms
- COVID-19, SARS-CoV-2, Algorithms, Artificial Intelligence, Female, Humans, Lung, Male, Middle Aged, Radiographic Image Interpretation, Computer-Assisted, Sensitivity and Specificity, Radiography, Thoracic, United States, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
