The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.
Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
Open accessAbstract
Abstract Machine learning methods offer great promise for fast and accurate detection and prognostication of coronavirus disease 2019 (COVID-19) from standard-of-care chest radiographs (CXR) and chest computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we consider all published papers and preprints, for the period from 1 January 2020 to 3 October 2020, which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. All manuscripts uploaded to bioRxiv, medRxiv and arXiv along with all entries in EMBASE and MEDLINE in this timeframe are considered. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher-quality model development and well-documented manuscripts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Much of the current evidence is based on retrospective analyses with limited prospective validation [19,28].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The success of deep learning in computer vision (CV) analysis has significantly advanced its clinical applications in comprehending biomedical images, including CT scans [88], positron emission tomography (PET)-CT scans [89], whole-slide images (WSIs) [90], and MRI scans [91].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [88]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Based on this study, we suggest that such a gold rush may give rise to un-engaged 'augmentation' which is highly risky from a learning and knowledge perspective for experts, AI companies, and consumers.”
From To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis (2022) · cites this paper as Roberts et al. 2021
In the citing paper
“Despite problems like the one discovered by Roberts et al. [195], the fact that ML systems process and analyze data differently from human beings can be a great benefit to science and society because these systems may be able to identify useful and innovative structures, properties, patterns, and relationships that human beings would not recognize.”
From The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool (2025) · cites this paper as [195]
In the citing paper
“Differences in image acquisition and processing may result in “shortcuts” where models take cues from non-clinical features.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“For example, COVID-19 prediction models have been demonstrated to learn which institution the images were obtained from rather than features pertaining to the underlying disease. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“The methodological flaws likely is a reflection of the lack of diversity in the development team to allow clinicians, data analysts and patient advocacy groups to work side-by-side throughout the AI life cycle. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“A similar result is summarized in a systematic review of COVID-19 prediction models on CT scans and Chest X-rays (CXRs) that found limited clinical utility due to methodological flaws and underlying biases stemming from small training data sets, data set variability, and limited integration of non-imaging data, among others.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Wang, D., Hu, B., Hu, C., Zhu, F., Liu, X., Zhang, J., Wang, B., Xiang, H., Cheng, Z., Xiong, Y., et al. (2020) · JAMA · doi:10.1001/jama.2020.1585 · Cited by 20,148
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Ai, T., Yang, Z., Hou, H., Zhan, C., Chen, C., Lv, W., Tao, Q., Sun, Z., & Xia, L. (2020) · Radiology · doi:10.1148/radiol.2020200642 · Cited by 5,793
Self-citationRadiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688
Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning
Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C. S., Liang, H., Baxter, S. L., McKeown, A., Yang, G., Wu, X., Yan, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.010 · Cited by 4,842
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR
Fang, Y., Zhang, H., Xie, J., Lin, M., Ying, L., Pang, P., & Ji, W. (2020) · Radiology · doi:10.1148/radiol.2020200432 · Cited by 3,340
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group† (2019) · Annals of Internal Medicine · doi:10.7326/m18-1376 · Cited by 2,984
COVID-19 and the cardiovascular system
Zheng, Y.-Y., Ma, Y.-T., Zhang, J.-Y., & Xie, X. (2020) · Nature Reviews Cardiology · doi:10.1038/s41569-020-0360-5 · Cited by 2,970
A deep learning algorithm using CT images to screen for Corona virus disease (COVID-19)
Wang, S., Kang, B., Ma, J., Zeng, X., Xiao, M., Guo, J., Cai, M., Yang, J., Li, Y., Meng, X., et al. (2021) · European Radiology · doi:10.1007/s00330-021-07715-1 · Cited by 1,694
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443
Shi, F., Wang, J., Shi, J., Wu, Z., Wang, Q., Tang, Z., He, K., Shi, Y., & Shen, D. (2021) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.2987975 · Cited by 1,404
Artificial intelligence–enabled rapid diagnosis of patients with COVID-19
Mei, X., Lee, H.-C., Diao, K.-Y., Huang, M., Lin, B., Liu, C., Xie, Z., Ma, Y., Robson, P. M., Chung, M., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0931-3 · Cited by 1,084
A deep learning algorithm using CT images to screen for Corona Virus Disease (COVID-19)Preprint
Wang, S., Kang, B., Ma, J., et al. (2020) · medRxiv · doi:10.1101/2020.02.14.20023028 · Cited by 1,035
Diagnosis of the Coronavirus disease (COVID-19): rRT-PCR or CT?
Long, C.-Q., Xu, H., Shen, Q., Zhang, X., Fan, B., Wang, C., Zeng, B., Li, Z., Li, X., & Li, H. (2020) · European Journal of Radiology · doi:10.1016/j.ejrad.2020.108961 · Cited by 987
COVID-CT-Dataset: A CT Scan Dataset about COVID-19Preprint
Yang, X., Zhao, J.-Y., Zhang, Y.-C., He, X.-H., & Xie, P.-T. (2020) · arXiv.org · doi:10.48550/arxiv.2003.13865 · Cited by 926
Ardakani, A. A., Kanafi, A. R., Acharya, U. R., Khadem, N., & Mohammadi, A. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103795 · Cited by 864
COVID-19 Image Data CollectionPreprint
Cohen, J., Morrison, P., & Dao, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.11597 · Cited by 722
Multi-task Deep Learning Based CT Imaging Analysis For COVID-19: Classification and Segmentation
Amyar, A., Modzelewski, R., Li, H., & Ruan, S. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104037 · Cited by 626
AI-assisted CT imaging analysis for COVID-19 screening: Building and deploying a medical AI system
Wang, B., Jin, S., Yan, Q., Xu, H., Luo, C. W., Wei, L., Zhao, W. Z., Hou, X., Ma, W., Xu, Z., et al. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106897 · Cited by 548
COVID-ResNet: A Deep Learning Framework for Screening of COVID19 from RadiographsPreprint
Farooq, M. A., & Hafeez, A. B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.14395 · Cited by 506
A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis
Wang, S., Zha, Y., Li, W., Wu, Q., Li, X., Niu, M., Wang, M., Qiu, X., Li, H., He, Y., et al. (2020) · European Respiratory Journal · doi:10.1183/13993003.00775-2020 · Cited by 493
Self-citationHeidari, M., Mirniaharikandehei, S., Khuzani, A. Z., Danala, G., Qiu, Y., & Zheng, B. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104284 · Cited by 423
New machine learning method for image-based diagnosis of COVID-19
Elaziz, M. A., Hosny, K. M., Salah, A., Darwish, M. M., Lu, S., & Sahlol, A. T. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235187 · Cited by 408
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392
Bai, H. X., Wang, R., Xiong, Z., Hsieh, B., Chang, K., Halsey, K., Tran, T. M. L., Choi, J. W., Wang, D.-C., Shi, L.-B., et al. (2020) · Radiology · doi:10.1148/radiol.2020201491 · Cited by 381
Shi, F., Xia, L., Shan, F., Song, B., Wu, D., Wei, Y., Yuan, H., Jiang, H., He, Y., Gao, Y., et al. (2021) · Physics in Medicine and Biology · doi:10.1088/1361-6560/abe838 · Cited by 379
Luz, E. J. D. S., Silva, P., Silva, R., Silva, L., Guimarães, J., Miozzo, G., Moreira, G., & Menotti, D. (2021) · Research on Biomedical Engineering · doi:10.1007/s42600-021-00151-6 · Cited by 370
Accurate Screening of COVID-19 Using Attention-Based Deep 3D Multiple Instance Learning
Han, Z., Wei, B., Hong, Y., Li, T., Cong, J., Zhu, X. L., Wei, H., & Zhang, W. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.2996256 · Cited by 328
Self-citationEstimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) DetectionPreprint
Ghoshal, B., & Tucker, A. (2020) · Brunel University Research Archive (BURA) (Brunel University London) · doi:10.48550/arxiv.2003.10769 · Cited by 327
Diagnosis of COVID-19 using CT scan images and deep learning techniques
Shah, V., Keniya, R., Shridharani, A., Punjabi, M., Shah, J., & Mehendale, N. D. (2021) · Emergency Radiology · doi:10.1007/s10140-020-01886-y · Cited by 320
Rahaman, M. M., Li, C., Yao, Y., Kulwa, F., Rahman, M. A., Wang, Q., Qi, S., Kong, F., Zhu, X., & Zhao, X. (2020) · Journal of X-Ray Science and Technology · doi:10.3233/xst-200715 · Cited by 311
Self-citationJin, S., Wang, B., Xu, H., et al. (2020) · medRxiv · doi:10.1101/2020.03.19.20039354 · Cited by 306
Albahri, O. S., Zaidan, A. A., Albahri, A. S., Zaidan, B. B., Abdulkareem, K. H., Al-Qaysi, Z. T., Alamoodi, A. H., Aleesa, A. M., Chyad, M. A., Alesa, R. M., et al. (2020) · Journal of Infection and Public Health · doi:10.1016/j.jiph.2020.06.028 · Cited by 287
Ko, H., Chung, H., Kang, W. S., Kim, K. W., Shin, Y., Kang, S., Lee, J. H., Kim, Y. J., Kim, N. Y., Jung, H., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/19569 · Cited by 279
A critic evaluation of methods for COVID-19 automatic detection from X-ray images
Maguolo, G., & Nanni, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.04.008 · Cited by 265
Predicting COVID-19 Pneumonia Severity on Chest X-ray With Deep Learning
Cohen, J., Dao, L., Roth, K., Morrison, P., Bengio, Y., Abbasi, A., Shen, B., Mahsa, H. K., Ghassemi, M., Li, H., et al. (2020) · Cureus · doi:10.7759/cureus.9448 · Cited by 261
Prior-Attention Residual Learning for More Discriminative COVID-19 Screening in CT Images
Wang, J., Bao, Y., Wen, Y., Lu, H., Luo, H., Yunfei, X., Li, X., Liu, C., & Qian, D. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.2994908 · Cited by 253
Yue, H., Yu, Q., Liu, C., Huang, Y., Jiang, Z., Shao, C., Zhang, H., Ma, B., Wang, Y., Xie, G., et al. (2020) · Annals of Translational Medicine · doi:10.21037/atm-20-3026 · Cited by 241
Ezzat, D., Hassanien, A. E., & Aboul-Ella, H. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106742 · Cited by 226
Unveiling COVID-19 from CHEST X-ray with deep learning: A hurdles race with small data
Tartaglione, E., Barbano, C. A., Berzovini, C., Calandri, M., & Grangetto, M. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17186933 · Cited by 211
Momentum contrastive learning for few-shot COVID-19 diagnosis from chest CT images
Chen, X., Yao, L., Zhou, T., Dong, J., & Zhang, Y. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107826 · Cited by 191
Wang, Z., Xiao, Y., Li, Y., Zhang, J., Lu, F., Hou, M., & Liu, X. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107613 · Cited by 190
Lassau, N., Ammari, S., Chouzenoux, É., Gortais, H., Hérent, P., Devilder, M., Soliman, S., Meyrignac, O., Talabard, M.-P., Lamarque, J.-P., et al. (2021) · Nature Communications · doi:10.1038/s41467-020-20657-4 · Cited by 187
AI-driven quantification, staging and outcome prediction of COVID-19 pneumonia
Chassagnon, G., Vakalopoulou, M., Battistella, E., Christodoulidis, S., Hoang-Thi, T.-N., Dangeard, S., Deutsch, É., André, F., Guillo, E., Halm, N., et al. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101860 · Cited by 179
Li, M., Arun, N., Gidwani, M., Chang, K., Deng, F. M., Little, B. P., Mendoza, D. P., Lang, M., Lee, S. I., O’Shea, A., et al. (2020) · Radiology Artificial Intelligence · doi:10.1148/ryai.2020200079 · Cited by 167
Huang, P., Lin, C. T., Li, Y., Tammemägi, M. C., Brock, M. V., Atkar-Khattra, S., Xu, Y., Hu, P., Mayo, J. R., Schmidt, H., et al. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30159-1 · Cited by 157
COVID-MobileXpert: On-Device COVID-19 Patient Triage and Follow-up using Chest X-rays
Li, X. S., Li, C., & Zhu, D. (2020) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9313217 · Cited by 129
Chen, X., Tang, Y., Mo, Y., Li, S., Lin, D., Yang, Z., Yang, Z., Sun, H., Qiu, J., Liao, Y., et al. (2020) · European Radiology · doi:10.1007/s00330-020-06829-2 · Cited by 127
Radiomics Analysis of Computed Tomography helps predict poor prognostic outcome in COVID-19
Wu, Q., Wang, S., Li, L., Wu, Q., Qian, W., Hu, Y., Li, L., Zhou, X., Ma, H., Li, H., et al. (2020) · Theranostics · doi:10.7150/thno.46428 · Cited by 127
Self-citationZhang, R., Tie, X., Qi, Z., Bevins, N., Zhang, C., Griner, D., Song, T. K., Nadig, J., Schiebler, M. L., Garrett, J. W., et al. (2020) · Radiology · doi:10.1148/radiol.2020202944 · Cited by 125
Interpretable artificial intelligence framework for COVID‑19 screening on chest X‑rays
Tsiknakis, N., Trivizakis, E., Vassalou, E. E., Papadakis, G. Z., Spandidos, D. A., Tsatsakis, A. M., Sánchez‑García, J., López‐González, R., Papanikolaou, N. K., Karantanas, A. H., et al. (2020) · Experimental and Therapeutic Medicine · doi:10.3892/etm.2020.8797 · Cited by 118
Model-based Prediction of Critical Illness in Hospitalized Patients with COVID-19
Schalekamp, S., Huisman, M., van Dijk, R. A., Boomsma, M. F., Jorge, P. F., de Boer, W. S., Herder, G. J. M., Bonarius, M., Groot, O. A., Jong, E., et al. (2020) · Radiology · doi:10.1148/radiol.2020202723 · Cited by 90
Wang, M., Chen, X., Huang, L., Xu, S., Qin, C., Liu, J., Cao, Y., Yu, P., Zhu, T., Zhu, H., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30199-0 · Cited by 86
Goncharov, M., Pisov, M., Shevtsov, A. E., Shirokikh, B., Kurmukov, A. G., Blokhin, I. A., Chernina, V., Solovev, A. V., Gombolevskiy, V. А., Morozov, S. P., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102054 · Cited by 85
Multi-task driven explainable diagnosis of COVID-19 using chest X-ray images
Malhotra, A., Mittal, S., Majumdar, P., Chhabra, S., Thakral, K., Mayank Vatsa, A., Singh, R., Chaudhury, S., Pudrod, A., & Agrawal, A. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108243 · Cited by 78
Integrative analysis for COVID-19 patient outcome prediction
Chao, H., Fang, X., Zhang, J., Homayounieh, F., Arru, C. D., Digumarthy, S. R., Babaei, R., Mobin, H. K., Mohseni, I., Saba, L., et al. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101844 · Cited by 77
Prediction models for diagnosis and prognosis in Covid-19
Sperrin, M., Grant, S. W., & Peek, N. B. (2020) · BMJ · doi:10.1136/bmj.m1464 · Cited by 74
Joint prediction and time estimation of COVID-19 developing severe symptoms using chest CT scan
Zhu, X., Song, B., Shi, F., Chen, Y., Hu, R., Gan, J., Zhang, W., Li, M., Wang, L., Gao, Y., et al. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101824 · Cited by 74
Self-citationCOVID-19 in CXR: From Detection and Severity Scoring to Patient Disease Monitoring
Frid-Adar, M., Amer, R., Gozes, O., Nassar, J., & Greenspan, H. K. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3069169 · Cited by 70 · cited 2×
“(1) feature reduction techniques in 52% of papers (23), (2) model validation in 61% of papers (27).”
InfluentialWei, W., Hu, X.-W., Cheng, Q., Zhao, Y.-M., & Ge, Y. (2020) · European Radiology · doi:10.1007/s00330-020-07012-3 · Cited by 67
Machine learning for COVID-19—asking the right questions
Bächtiger, P., Peters, N. S., & Walsh, S. L. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30162-x · Cited by 65
Xu, M. J., Ouyang, L., Gao, Y., Chen, Y., Yu, T., Li, Q., Sun, K., Bao, F. S., Safarnejad, L., Wen, J., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/25535 · Cited by 64
Qin, L., Yang, Y., Cao, Q., Cheng, Z., Wang, X., Sun, Q., Yan, F., Qu, J., & Yang, W. (2020) · European Radiology · doi:10.1007/s00330-020-07022-1 · Cited by 57
Development and Validation of an Automated Radiomic CT Signature for Detecting COVID-19
Guiot, J., Vaidyanathan, A., Deprez, L., Zerka, F., Danthine, D., Frix, A.-N., Thys, M., Henket, M. A., Canivet, G., Mathieu, S., et al. (2020) · Diagnostics · doi:10.3390/diagnostics11010041 · Cited by 57
Zhang, R., Guo, Z., Sun, Y., Lu, Q., Xu, Z., Yao, Z., Duan, M., Liu, S., Ren, Y., Huang, L., et al. (2020) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-020-00393-5 · Cited by 56
Acar, E., Şahin, E., & Yılmaz, İ. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06344-5 · Cited by 52
Tamal, M., AlShammari, M. M., Alabdullah, M., Hourani, R., Alola, H. A., & Hegazi, T. M. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115152 · Cited by 48 · cited 1×
“(1) feature reduction techniques in 52% of papers (23), (2) model validation in 61% of papers (27).”
Any unique image biomarkers associated with COVID-19?
Pu, J., Leader, J. K., Bandos, A. I., Shi, J., Du, P., Yu, J., Yang, B., Shi, K., Guo, Y., Field, J. B., et al. (2020) · European Radiology · doi:10.1007/s00330-020-06956-w · Cited by 45
Chen, Y., Wang, Y., Zhang, Y., Zhang, N., Zhao, S., Zeng, H., Deng, W., Huang, Z., Liu, S., & Song, B. (2020) · International Journal of Medical Sciences · doi:10.7150/ijms.48432 · Cited by 43
Artificial intelligence techniques for Containment COVID-19 Pandemic: A Systematic ReviewPreprint
Alabool, H. M., Alarabiat, D., Abualigah, L., Habib, M., Al-Khasawneh, A., Alshinwan, M., & Shehab, M. (2020) · doi:10.21203/rs.3.rs-30432/v1 · Cited by 33
Zokaeinikoo, M., Kazemian, P., Mitra, P., & Kumara, S. R. T. (2021) · ACM Transactions on Management Information Systems · doi:10.1145/3466690 · Cited by 33
Development and validation of an automated radiomic CT signature for detecting COVID-19Preprint
Guiot, J., Vaidyanathan, A., Deprez, L., et al. (2020) · medRxiv · doi:10.1101/2020.04.28.20082966 · Cited by 32
AI-Driven CT-based quantification, staging and short-term outcome prediction of COVID-19 pneumoniaPreprint
Chassagnon, G., Vakalopoulou, M., Battistella, E., Christodoulidis, S., Hoang-Thi, T., Dangeard, S., Deutsch, É., André, F., Guillo, E., Halm, N., et al. (2020) · medRxiv · doi:10.1101/2020.04.17.20069187 · Cited by 31
Machine learning automatically detects COVID-19 using chest CTs in a large multicenter cohort
Barbosa, E. J. M., Georgescu, B., Chaganti, S., Bastarrika, G., Cabrero, J. B., Chabin, G., Flohr, T. G., Greniér, P., Grbić, S., Gupta, N., et al. (2021) · European Radiology · doi:10.1007/s00330-021-07937-3 · Cited by 31
Diagnosis of COVID-19 using CT scan images and deep learning techniquesPreprint
Shah, V., Keniya, R., Shridharani, A., et al. (2020) · medRxiv · doi:10.1101/2020.07.11.20151332 · Cited by 31
Xie, C., Ng, M., Ding, J., Leung, S. T., Lo, C. S. Y., Wong, H. Y. F., & Vardhanabhuti, V. (2020) · European Journal of Radiology Open · doi:10.1016/j.ejro.2020.100271 · Cited by 31
Zheng, Y., Xiao, A., Yu, X., Zhao, Y., Lu, Y., Li, X., Mei, N., She, D., Wang, D., Geng, D., et al. (2020) · Korean Journal of Radiology · doi:10.3348/kjr.2020.0485 · Cited by 29
A web-based Diagnostic Tool for COVID-19 Using Machine Learning on Chest Radiographs (CXR)Preprint
Kana, E. B. G., Kana, M. G. Z., Kana, A. F. D., & Kenfack, R. H. A. (2020) · medRxiv · doi:10.1101/2020.04.21.20063263 · Cited by 25
Wang, X., Hu, X., Tan, W., Mazzone, P. J., Mireles‐Cabodevila, E., Han, X., Huang, P., Hu, W., Dweik, R. A., & Cheng, Z. (2020) · American Journal of Roentgenology · doi:10.2214/ajr.20.24044 · Cited by 25
Li, M., Arun, N., Aggarwal, M., Gupta, S., Singh, P., Little, B. P., Mendoza, D. P., Corradi, G. C. A., Takahashi, M. S., Ferraciolli, S. F., et al. (2022) · Medicine · doi:10.1097/md.0000000000029587 · Cited by 25
Acar, E., Şahin, E., & Yılmaz, İ. (2020) · medRxiv · doi:10.1101/2020.06.12.20129643 · Cited by 17
Zokaeinikoo, M., Kazemian, P., Mitra, P., et al. (2020) · medRxiv · doi:10.1101/2020.05.24.20111922 · Cited by 16
Xu, M. J., Ouyang, L., Gao, Y., et al. (2020) · medRxiv · doi:10.1101/2020.08.18.20176776 · Cited by 15
Ramtohul, T., Cabel, L., Paolettí, X., Chiche, L., Moreau, P., Noret, A., Vuagnat, P., Chérel, P., Tardivon, A. A., Cottu, P. H., et al. (2020) · Frontiers in Oncology · doi:10.3389/fonc.2020.01560 · Cited by 14
Performance and Robustness of Machine Learning-based Radiomic COVID-19 Severity PredictionPreprint
Yip, S. S. F., Klaneček, Ž., Naganawa, S., Kim, J., Studen, A., Rivetti, L., & Jeraj, R. (2020) · medRxiv · doi:10.1101/2020.09.07.20189977 · Cited by 12
Early Screening of SARS-CoV-2 by Intelligent Analysis of X-Ray ImagesPreprint
Gil, D., Díaz-Chito, K., Sánchez, C., & Hernández-Sabaté, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.13928 · Cited by 11 · cited 1×
“(1) Feature reduction techniques in 52% of papers (23) (2) Model validation in 61% of papers (27)”
BackgroundLassau, N., Ammari, S., Chouzenoux, E., et al. (2020) · medRxiv · doi:10.1101/2020.05.14.20101972 · Cited by 10
Ghosh, B., Kumar, N., Sadhu, A. K., Ghosh, N., Mitra, P., & Chatterjee, J. (2020) · medRxiv · doi:10.1101/2020.07.13.20152231 · Cited by 9
CovMUNET: A Multiple Loss Approach towards Detection of COVID-19 from Chest X-rayPreprint
Sayyed, A. Q. M. S., Saha, D., & Hossain, A. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14318 · Cited by 7
Bararia, A., Ghosh, A., Bose, C., & Bhar, D. (2020) · medRxiv · doi:10.1101/2020.09.07.20189852 · Cited by 6
Was there COVID-19 back in 2012? Challenge for AI in Diagnosis with Similar IndicationsPreprint
Banerjee, I., Sinha, P., Purkayastha, S., Mashhaditafreshi, N., Tariq, A., Jeong, J., Trivedi, H., & Gichoya, J. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.13262 · Cited by 5
Research works printed but not linked: 7
- Wong, H. Y. F. et al. Frequency and distribution of chest radiographic findings in COVID-19 positive patients. Radiology 296, 201160 (2019).
- Bassi, P. R. A. S. & Attux, R. A deep convolutional neural network for COVID-19 detection using chest X-rays. Preprint at http://arxiv.org/abs/2005.01578 (2020). · arXiv
- Amyar, A., Modzelewski, R. & Ruan, S. Multi-task deep learning based CT imaging analysis for COVID-19: classification and segmentation. Preprint at medRxiv https://doi.org/10.1101/2020.04.16.2006470 (2020). · DOI
- Goodfellow, I. J. et al. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems Vol. 2 2672–2680 (MIT Press, 2014).
- Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations ICLR 2017 (ICLR, 2017).
- Covidence Systematic Review Software v2486 6284093b (Veritas Health Innovation, 2020); https://www.covidence.org/
- Aviles-Rivero, A. I., Sellars, P., Schönlieb, C.-B. & Papadakis, N. GraphXCOVID: explainable deep graph diffusion pseudo-labelling for identifying COVID-19 on chest X-rays. Preprint at https://arxiv.org/abs/2010.00378 (2020). · arXiv
Cited by
Sorted by citations
A guide to machine learning for biologists
Greener, J. G., Kandathil, S. M., Moffat, L., & Jones, D. T. (2022) · Nature Reviews Molecular Cell Biology · doi:10.1038/s41580-021-00407-0 · Cited by 2,127
Computational approaches streamlining drug discovery
Sadybekov, A., & Katritch, V. (2023) · Nature · doi:10.1038/s41586-023-05905-z · Cited by 1,424
“It has been shown that DL models, especially based on limited datasets lacking negative data, are prone to overtraining and spurious performance, sometimes leading to whole classes of models deemed ‘useless’ 88 or severely biased by subjective factors defining the training dataset 89 .”
BackgroundLeakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
Recent advances and clinical applications of deep learning in medical image analysis
Chen, X., Wang, X., Zhang, K., Fung, K.-M. A., Thai, T., Moore, K. N., Mannel, R. S., Liu, H., Zheng, B., & Qiu, Y. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102444 · Cited by 1,180
Self-citationRedefining Radiology: A Review of Artificial Intelligence Integration in Medical Imaging
Najjar, R. (2023) · Diagnostics · doi:10.3390/diagnostics13172760 · Cited by 1,025
“…mathematical mirroring of reality, depends not only on their training datasets and precision but also on calibration in interpreting medical images depends on comprehensive datasets that accurately reflect varied patient demographics, including age, sex, ethnicity, and disease stages [94,95].”
Background
Metadata
- Authors
- Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., Weir-McCall, J. R., Teng, Z., Gkrania-Klotsas, E., AIX-COVNET, Ruggiero, A., Korhonen, A., Jefferson, E., Ako, E., Langs, G., Gozaliasl, G., Yang, G., Prosch, H., Preller, J., Stanczuk, J., Tang, J., Hofmanninger, J., Babar, J., Sánchez, L. E., Thillai, M., Gonzalez, P. M., Teare, P., Zhu, X., Patel, M., Cafolla, C., Azadbakht, H., Jacob, J., Lowe, J., Zhang, K., Bradley, K., Wassin, M., Holzer, M., Ji, K., Ortet, M. D., Ai, T., Walton, N., Lio, P., Stranks, S., Shadbahr, T., Lin, W., Zha, Y., Niu, Z., Rudd, J. H. F., Sala, E., & Schönlieb, C.-B.
- Affiliations
- AstraZeneca (United Kingdom) (GB)
- University of Cambridge (GB)
- University of Manchester (GB)
- Royal Papworth Hospital (GB)
- University of Dundee (GB)
- Royal Brompton Hospital (GB)
- Medical University of Vienna (AT)
- University of Helsinki (FI)
- Imperial College London (GB)
- Boehringer Ingelheim (Germany) (DE)
- Addenbrooke's Hospital (GB)
- University College London (GB)
- Macau University of Science and Technology (MO)
- Tongji Hospital (CN)
- Renmin Hospital of Wuhan University (CN)
- Corresponding author
- Roberts, Michael
- ORCID
- Roberts, Michael
- Thorpe, Matthew
- Gilbey, Julian
- Yeung, Michael
- Ursprung, Stephan
- McCague, Cathal
- Weir-McCall, Jonathan R.
- Gkrania-Klotsas, Effrossyni
- Ako, Emmanuel
- Langs, Georg
- Gozaliasl, Ghassem
- Yang, Guang
- Prosch, Helmut
- Preller, Jacobus
- Azadbakht, Hojjat
- Jacob, Joseph
- Ji, Kangyu
- Walton, Nicholas
- Stranks, Samuel
- Shadbahr, Tolou
- Rudd, James H. F.
- Sala, Evis
- Schönlieb, Carola-Bibiane
- Venue
- Nature Machine Intelligence
- Published
- 2021-03-15
- Volume
- 3
- Issue
- 3
- Pages
- 199-217
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W3119527628
- Cited by
- 1,044
- References
- 83
- Citations by year
- 2026: 203 · 2025: 96 · 2024: 143 · 2023: 248 · 2022: 224 · 2021: 130
- FWCI
- 98.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Machine Intelligence (publishedVersion, cc-by)
- Discovery Research Portal (University of Dundee) (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- elib (German Aerospace Center) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Työväentutkimus Vuosikirja (submittedVersion)
- arXiv (Cornell University)
- Funders
- AstraZeneca
- European Commission (777826)
- Medical Research Council (1966157, MR/S010351/1)
- Engineering and Physical Sciences Research Council (EP/T017961/1, EP/N014588/1, EP/S026045/1)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- COVID-19 pandemic, machine learning, chest radiograph, chest CT, COVID-19 detection, outcome prediction, clinical utility, model validation, methodological errors, bias, research recommendations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
