The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.
Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
Open accessAbstract
Abstract Machine learning methods offer great promise for fast and accurate detection and prognostication of coronavirus disease 2019 (COVID-19) from standard-of-care chest radiographs (CXR) and chest computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we consider all published papers and preprints, for the period from 1 January 2020 to 3 October 2020, which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. All manuscripts uploaded to bioRxiv, medRxiv and arXiv along with all entries in EMBASE and MEDLINE in this timeframe are considered. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher-quality model development and well-documented manuscripts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Much of the current evidence is based on retrospective analyses with limited prospective validation [19,28].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The success of deep learning in computer vision (CV) analysis has significantly advanced its clinical applications in comprehending biomedical images, including CT scans [88], positron emission tomography (PET)-CT scans [89], whole-slide images (WSIs) [90], and MRI scans [91].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [88]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Differences in image acquisition and processing may result in “shortcuts” where models take cues from non-clinical features.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“Based on this study, we suggest that such a gold rush may give rise to un-engaged 'augmentation' which is highly risky from a learning and knowledge perspective for experts, AI companies, and consumers.”
From To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis (2022) · cites this paper as Roberts et al. 2021
In the citing paper
“Despite problems like the one discovered by Roberts et al. [195], the fact that ML systems process and analyze data differently from human beings can be a great benefit to science and society because these systems may be able to identify useful and innovative structures, properties, patterns, and relationships that human beings would not recognize.”
From The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool (2025) · cites this paper as [195]
In the citing paper
“For example, COVID-19 prediction models have been demonstrated to learn which institution the images were obtained from rather than features pertaining to the underlying disease. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“The methodological flaws likely is a reflection of the lack of diversity in the development team to allow clinicians, data analysts and patient advocacy groups to work side-by-side throughout the AI life cycle. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“A similar result is summarized in a systematic review of COVID-19 prediction models on CT scans and Chest X-rays (CXRs) that found limited clinical utility due to methodological flaws and underlying biases stemming from small training data sets, data set variability, and limited integration of non-imaging data, among others.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Wang, D., Hu, B., Hu, C., Zhu, F., Liu, X., Zhang, J., Wang, B., Xiang, H., Cheng, Z., Xiong, Y., et al. (2020) · JAMA · doi:10.1001/jama.2020.1585 · Cited by 20,148
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Ai, T., Yang, Z., Hou, H., Zhan, C., Chen, C., Lv, W., Tao, Q., Sun, Z., & Xia, L. (2020) · Radiology · doi:10.1148/radiol.2020200642 · Cited by 5,793
Self-citationRadiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688
Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning
Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C. S., Liang, H., Baxter, S. L., McKeown, A., Yang, G., Wu, X., Yan, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.010 · Cited by 4,842
Self-citation
Research works printed but not linked: 7
- Wong, H. Y. F. et al. Frequency and distribution of chest radiographic findings in COVID-19 positive patients. Radiology 296, 201160 (2019).
- Bassi, P. R. A. S. & Attux, R. A deep convolutional neural network for COVID-19 detection using chest X-rays. Preprint at http://arxiv.org/abs/2005.01578 (2020). · arXiv
- Amyar, A., Modzelewski, R. & Ruan, S. Multi-task deep learning based CT imaging analysis for COVID-19: classification and segmentation. Preprint at medRxiv https://doi.org/10.1101/2020.04.16.2006470 (2020). · DOI
- Goodfellow, I. J. et al. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems Vol. 2 2672–2680 (MIT Press, 2014).
- Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations ICLR 2017 (ICLR, 2017).
- Covidence Systematic Review Software v2486 6284093b (Veritas Health Innovation, 2020); https://www.covidence.org/
- Aviles-Rivero, A. I., Sellars, P., Schönlieb, C.-B. & Papadakis, N. GraphXCOVID: explainable deep graph diffusion pseudo-labelling for identifying COVID-19 on chest X-rays. Preprint at https://arxiv.org/abs/2010.00378 (2020). · arXiv
Cited by
Sorted by citations
Items 201–300 of 1,000
Multimodal explainability via latent shift applied to COVID-19 stratification
Guarrasi, V., Tronchin, L., Albano, D., Faiella, E., Fazzini, D., Santucci, D., & Soda, P. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110825 · Cited by 31
“For these reasons, several authors indicated that CXR imaging fit well with the needs of COVID-19 pandemic [21].”
Advancing Medical Imaging Research Through Standardization
Jeon, K., Park, W. Y., Kahn, C. E., Nagy, P. G., You, S. C., & Yoon, S. H. (2024) · Investigative Radiology · doi:10.1097/rli.0000000000001106 · Cited by 34
Fazzari, M. J., Guerra, M. M., Salmon, J. E., & Kim, M. (2022) · Lupus Science & Medicine · doi:10.1136/lupus-2022-000769 · Cited by 35
Comparative performances of machine learning algorithms in radiomics and impacting factors
Decoux, A., Duron, L., Habert, P., Roblot, V., Arsovic, E., Chassagnon, G., Arnoux, A., & Fournier, L. S. (2023) · Scientific Reports · doi:10.1038/s41598-023-39738-7 · Cited by 36
“The sarcopenia dataset was a set of CT images from a multicentric database 5 in which ROIs were defined on psoas and posterior muscles at L3 level to quantify muscle surface.”
MethodRani, G., Misra, A., Dhaka, V. S., Buddhi, D., Sharma, R. K., Zumpano, E., & Vocaturo, E. (2022) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2022.200148 · Cited by 30
Chen, Y., Wan, Y., & Pan, F. (2023) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-023-00801-4 · Cited by 32
“CoAtNet‐0‐rw Although CNN is still the predominant network in computer vision, Transformer has shown a powerful performance potential since its birth [6].”
BackgroundInfluentialMachine learning for healthcare that matters: Reorienting from technical novelty to equitable impact
Balagopalan, A., Baldini, I., Celi, L. A., Gichoya, J. W., McCoy, L. G., Naumann, T., Shalit, U., van der Schaar, M., & Wagstaff, K. L. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000474 · Cited by 34
“Models are too often developed without appropriate scrutiny of underlying data and subsequent outputs [14,21].”
BackgroundInfluentialLi, W., & Wang, J. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102643 · Cited by 28
External validation of Machine Learning models for COVID-19 detection based on Complete Blood Count
Campagner, A., Carobene, A., & Cabitza, F. (2021) · Health Information Science and Systems · doi:10.1007/s13755-021-00167-3 · Cited by 29
“Indeed, lack of external validation has recently been noted in [29], together with lack of reproducibility [3, 36], as being one of the main challenges to the real-world adoption of MLbased approaches for COVID-19 diagnosis.”
BackgroundMethodComplex Relationship Between Artificial Intelligence and CT Radiation Dose
Gupta, R. V., Kalra, M. K., Ebrahimian, S., Kaviani, P., Primak, A. N., Bizzo, B. C., & Dreyer, K. J. (2021) · Academic Radiology · doi:10.1016/j.acra.2021.10.024 · Cited by 29
Explainable Artificial Intelligence for COVID-19 Diagnosis Through Blood Test Variables
Thimoteo, L. M., Vellasco, M. M. B. R., Amaral, J. L. M., Figueiredo, K., Yokoyama, C. L., & Marques, E. (2022) · Journal of Control Automation and Electrical Systems · doi:10.1007/s40313-021-00858-y · Cited by 31
“By analyzing Roberts et al. (2021) and Mohammad-Rahimi et al. (2021), it is clear that most of the approaches make use of imaging data and sophisticated deep learning and computer vision techniques.”
Seedat, N., Imrie, F., & van der Schaar, M. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3345805 · Cited by 49
“Hence, while there was an illusion of reliable performance in detecting Covid-19, the reality is that the data mismatch transformed the model into an adult vs child detector [66].”
BackgroundDeep Morphology Learning Enhances Ex Vivo Drug Profiling-Based Precision Medicine
Heinemann, T., Kornauth, C., Severin, Y., Vladimer, G. I., Pemovska, T., Hadzijusufovic, E., Agis, H., Krauth, M.-T., Sperr, W. R., Valent, P., et al. (2022) · Blood Cancer Discovery · doi:10.1158/2643-3230.bcd-21-0219 · Cited by 28
“Furthermore, by training the deep learning on single-cell classification and deriving the clinical treatment recommendations from the resulting drug-screen analysis, rather than directly training the network on the identification of effective treatments, we avoid overfittingassociated limitations that have hampered th…”
BackgroundRoss, J., Hammouche, S., Chen, Y., Rockall, A., Alabed, S., Chen, M., Dwivedi, K., Fascia, D., Greenhalgh, R., Hall, M., et al. (2024) · Clinical Radiology · doi:10.1016/j.crad.2024.01.026 · Cited by 31
Wyatt, M., Radford, B., Callow, N., Bennamoun, M., & Hickey, S. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13841 · Cited by 31
Ma, R., Hu, J., Sari, H., Xue, S., Mingels, C., Viscione, M., Kandarpa, V. S. S., Li, W. B., Visvikis, D., Qiu, R., et al. (2022) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-022-05861-2 · Cited by 28
“A critical concern for AI 3 development is its reproducibility and extensibility to complexity in real applications [27].”
BackgroundArmato, S. G., Drukker, K., & Hadjiiski, L. M. (2023) · British Journal of Radiology · doi:10.1259/bjr.20221152 · Cited by 31
A Unified Framework on Generalizability of Clinical Prediction Models
Wan, B., Caffo, B. S., & Vedula, S. S. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.872720 · Cited by 37
“For example, only a few among 62 studies developing CPMs for diagnosis or prognosis of coronavirus disease 2019 (Covid-19) were assessed to be at low risk of bias based on study design (Roberts et al., 2021).”
BackgroundBridge, C. P., Gorman, C., Pieper, S., Doyle, S., Lennerz, J. K., Kalpathy–Cramer, J., Clunie, D., Fedorov, A., & Herrmann, M. D. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00683-y · Cited by 30
“, the conversion of a given research output into standard DICOM representation, is generally not possible, since important contextual information is lost along the way [48].”
Webb, E., & Scott, P. J. H. (2021) · PET Clinics · doi:10.1016/j.cpet.2021.06.012 · Cited by 27
Deep Learning for COVID-19 Diagnosis from CT Images
Loddo, A., Pili, F., & Ruberto, C. D. (2021) · Applied Sciences · doi:10.3390/app11178227 · Cited by 27
“In response to the pandemic, researchers have rushed to develop models using artificial intelligence (AI), particularly machine learning, to support clinicians [7].”
BackgroundInfluentialCalderon-Ramirez, S., Yang, S., Elizondo, D. A., & Moemeni, A. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.108983 · Cited by 27
“This dataset combination has been extensively used for training Convolutional Neural Network (CNN) based models to detect COVID-19, and leads to deceptive bias in both the test and training model data [50].”
DeceFL: a principled fully decentralized federated learning framework
Yuan, Y., Liu, J., Jin, D., Yue, Z., Yang, T., Chen, R., Wang, M., Xu, L., Hua, F., Guo, Y., et al. (2022) · National Science Open · doi:10.1360/nso/20220043 · Cited by 29
“This has become a well-known bottleneck in AI applications, especially medical AI [6].”
BackgroundAbbasi, F., Adatorwovor, R., Davarpanah, M. A., Mansoori, Y., Hajiani, M., Azodi, F., Sefidbakht, S., Davoudi, S., Rezaei, F., Mohammadmoradi, S., et al. (2022) · Journal of the Endocrine Society · doi:10.1210/jendso/bvac017 · Cited by 27
“However, the current literature has limitations in terms of diagnosis and prognosis determination for pneumonia due to COVID-19 by applying chest radiographs and CT scans [43].”
Gusev, A. V., Vladzymyrskyy, A. V., Sharova, D. E., Arzamasov, K. M., & Hramov, A. E. (2022) · Digital Diagnostics · doi:10.17816/dd107367 · Cited by 27
“Вместе с этим недавние публикации показали, что для широкого внедрения ИИ-систем в практическом здравоохранении нужны дополнительные исследования и разработки, в том числе для обеспечения независимой клинической проверки и оценки экономической эффективности [2, 3].”
Machine learning with multimodal data for COVID-19
Chen, W., Sá, R. C., Bai, Y., Napel, S. A., Gevaert, O., Lauderdale, D. S., & Giger, M. L. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e17934 · Cited by 27
“Reviews of general ML development in COVID-19 can be found in the literature [12–14]; however, these reviews either focus on imaging based studies [13,14] or include models based on different types of data but not integration of multimodal data [12].”
Data-related concepts for artificial intelligence education in K-12
Olari, V., & Romeike, R. (2024) · Computers and Education Open · doi:10.1016/j.caeo.2024.100196 · Cited by 33
“AI curricula also discuss whether the data is effectively represented by the model, described by concepts such as data fit [13], overfitting and underfitting [43,85].”
BackgroundCalle, P., Bates, A., Reynolds, J. C., Liu, Y., Cui, H., Ly, S., Wang, C., Zhang, Q., de Armendi, A. J., Shettar, S. S., et al. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.109063 · Cited by 26
Laino, M. E., Ammirabile, A., Posa, A., Cancian, P., Shalaby, S. Y., Savevski, V., & Neri, E. (2021) · Diagnostics · doi:10.3390/diagnostics11081317 · Cited by 27
“, QUADAS, CLAIM, RQS) and checklists to identify these weaknesses [138].”
BackgroundMethodInfluentialSadik, F., Dastider, A. G., Subah, M. R., Mahmud, T., & Fattah, S. A. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105806 · Cited by 25
Generalizable disease detection using model ensemble on chest X-ray images
Abad, M., Casas-Roma, J., & Prados, F. (2024) · Scientific Reports · doi:10.1038/s41598-024-56171-6 · Cited by 30
Tracking and predicting COVID-19 radiological trajectory on chest X-rays using deep learning
Gourdeau, D., Potvin, O., Archambault, P. M., Chartrand‐Lefebvre, C., Dieumegarde, L., Forghani, R., Gagné, C., Hains, A., Hornstein, D., Le, H. B. Q., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-09356-w · Cited by 27
“The issue of dataset bias is very problematic in most COVID-19 prognosis studies as pointed by 32,33 , which propose as a solution to identify a very specific use-case and then proceed to collect a representative dataset.”
BackgroundShiri, I., Salimi, Y., Saberi, A., Pakbin, M., Hajianfar, G., Avval, A. H., Sanaat, A., Akhavanallaf, A., Mostafaei, S., Mansouri, Z., et al. (2021) · medRxiv · doi:10.1101/2021.12.07.21267367 · Cited by 26
Gillette, K. K., Gsell, M. A. F., Nagel, C., Bender, J., Winkler, B., Williams, S. E., Bär, M., Schaeffter, T., Dössel, O., Plank, G., et al. (2023) · Scientific Data · doi:10.1038/s41597-023-02416-4 · Cited by 26
“202117 λT,i, i ∈ [1,2] [-2, 2] Pishchulin et al.”
Chau, M. T. (2024) · Journal of medical imaging and radiation sciences · doi:10.1016/j.jmir.2024.101733 · Cited by 30
Johri, S., Goyal, M., Jain, S., Baranwal, M., Kumar, V., & Upadhyay, R. (2021) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.22613 · Cited by 25
“42,43 system, which can be installed in hospitals having limited resources for patient testing.”
BackgroundTraining Deep Networks from Zero to Hero: avoiding pitfalls and going beyond
Ponti, M. A., dos Santos, F. P., Ribeiro, L. S. F., & Cavallari, G. B. (2021) · SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI) · doi:10.1109/sibgrapi54419.2021.00011 · Cited by 26
Chen, S., Liu, M., Pan, D., Deng, J., Yuan, Y., Cheng, X., Xie, T., Xie, L., Zhang, W., Gong, H., et al. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3197666 · Cited by 26
“For the dataset, as with most current classification and prediction models of COVID-19, the diagnosis and prediction models developed from radiological imaging data are limited by the quality of the dataset [40].”
MethodAl‐Yousif, N., Komanduri, S., QURASHI, H. M. S., Korzhuk, A., Lawal, H. O., Abourizk, N., Schaefer, C., Mitchell, K. J., Dietz, C. M., Hughes, E. K., et al. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-066626 · Cited by 25
Towards precision medicine: Omics approach for COVID-19
Cen, X., Wang, F., Huang, X., Jovic, D., Dubee, F., Yang, H., & Li, Y. (2023) · Biosafety and Health · doi:10.1016/j.bsheal.2023.01.002 · Cited by 26
“Model interpretation is an important challenge for deep ‐ learning ‐ based COVID ‐ 19 diagnosis systems [85].”
Orbit-to-ground framework to decode and predict biosignature patterns in terrestrial analogues
Warren‐Rhodes, K., Cabrol, N. A., Phillips, M., Tebes-Cayo, C., Kalaitzis, F., Ayma, D., Demergasso, C. S., Chong, G., Lee, K. C. Y., Hinman, N. W., et al. (2023) · Nature Astronomy · doi:10.1038/s41550-022-01882-x · Cited by 25
Proactive vs Reactive Machine Learning in Health Care: Lessons From the COVID-19 Pandemic
Luo, Y., Wunderink, R. G., & Lloyd-Jones, D. (2022) · JAMA · doi:10.1001/jama.2021.24935 · Cited by 24
Artificial intelligence agents in cancer research and oncology
Truhn, D., Azizi, S., Zou, J., Cerdá-Alberich, L., Mahmood, F., & Kather, J. N. (2026) · Nature reviews. Cancer · doi:10.1038/s41568-025-00900-0 · Cited by 29
Chou, A., Espin, A. T., Kyritsis, N., Huie, J. R., Khatry, S., Funk, J., Hay, J., Lofgreen, A., Shah, R. R., McCann, C., et al. (2022) · PLoS ONE · doi:10.1371/journal.pone.0265254 · Cited by 24
“However, clinical utility requires ML models to be interpretable for biological mechanisms, verifiable by clinicians, and methodologically and inferentially reproducible [15, 16].”
BackgroundAlabed, S., Maiter, A., Salehi, M., Mahmood, A., Daniel, S., Jenkins, S., Goodlad, M., Sharkey, M. J., Mamalakis, M., Rakocevic, V., et al. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.956811 · Cited by 25
“Furthermore, given that AI studies in chest imaging have shown similar shortcomings in reporting quality (14), our findings may be more broadly relevant to AI studies in medical imaging.”
ResultChamberlin, J., Aquino, G. J., Nance, S., Wortham, A., Leaphart, N., Paladugu, N., Brady, S., Baird, H., Fiegel, M. J., Fitzpatrick, L., et al. (2022) · BMC Infectious Diseases · doi:10.1186/s12879-022-07617-7 · Cited by 24
“…by low sample-size, unclear origins of training data (including public datasets with poorly annotated images), lack of a real world testing cohort, and absence of follow-up with clinical outcomes [14]. dCNNs are also notorious for having “black box” outputs and a lack of interpretability [19].”
BackgroundMethodWang, Y., Jiang, C., Wu, Y., Lv, T., Sun, H., Liu, Y., Li, L., & Pan, X. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3205167 · Cited by 24
“Medical imaging technologies, such as Computed Tomography (CT) or Chest X-ray (CXR), can play a vital role in diagnosing COVID-19 patients, especially in the cases of infected children and pregnant women [6], [42].”
MethodGarcía‐Jacas, C. R., García‐González, L. A., Martínez-Rios, F., Tapia-Contreras, I. P., & Brizuela, C. A. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac428 · Cited by 25
Xue, S., & Abhayaratne, C. (2023) · IEEE Access · doi:10.1109/access.2023.3260632 · Cited by 24
“Although in general there have been many published papers using deep learning for CT image analysis, very few papers proposed reproducible methods that follow the best practice for deep learning models and justify their applicability in real-world in-vivo scenarios [19].”
BackgroundAutomated COVID-19 detection with convolutional neural networks
Dumakude, A., & Ezugwu, A. E.-S. (2023) · Scientific Reports · doi:10.1038/s41598-023-37743-4 · Cited by 27
“In response to the outbreak, researchers have expedited the development of COVID-19 detection models utilizing artificial intelligence to assist clinicians 2 .”
BackgroundTürkbey, B., Huisman, H. J., Fedorov, A., Macura, K. J., Margolis, D., Panebianco, V., Oto, A., Schoots, I. G., Siddiqui, M. M., Moore, C. M., et al. (2025) · Radiology · doi:10.1148/radiol.240140 · Cited by 23
Schiaffino, S., Codari, M., Cozzi, A., Albano, D., Alì, M., Arioli, R., Avola, E., Bnà, C., Cariati, M., Carriero, S., et al. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11060501 · Cited by 23
Rapid Nanoplasmonic-Enhanced Detection of SARS-CoV-2 and Variants on DNA Aptamer Metasurfaces
Torun, H., Bilgin, B., İlgü, M., Batur, N., Öztürk, M., Barlas, T., Güney-Esken, G., Yanık, C., Çelik, S., Doğan, Ö., et al. (2023) · Advanced Devices & Instrumentation · doi:10.34133/adi.0008 · Cited by 23
Coppock, H., Nicholson, G., Kiskin, I., Koutra, V., Baker, K., Budd, J., Payne, R., Karoune, E., Hurley, D., Titcomb, A., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-023-00773-8 · Cited by 35
“If a study’s enrolment bias is unaddressed and shared across both training and held-out test sets, a classifier that seems to perform well may not generalize to other datasets 20,21 .”
AI in Combating the COVID-19 Pandemic
Cao, L. (2022) · IEEE Intelligent Systems · doi:10.1109/mis.2022.3164313 · Cited by 22
Purcell, C. R., Walsh, A., Colefax, A. P., & Butcher, P. A. (2022) · Frontiers in Marine Science · doi:10.3389/fmars.2022.981897 · Cited by 22
Shulha, M., Hovdebo, J., D’Souza, V., Thibault, F., & Harmouche, R. (2024) · JMIR Formative Research · doi:10.2196/50475 · Cited by 22
“Barriers to adoption in health care include data quality; data bias; and lack of proper validation, reproducibility, and transparency [2].”
BackgroundArtificial Intelligence and the Practice of Neurology in 2035
Jones, D. T., & Kerber, K. A. (2022) · Neurology · doi:10.1212/wnl.0000000000013200 · Cited by 25
“” 26 Advanced “ multi-omics ” technologies combined with ML have most widely been utilized in oncology with great promise for cancer detection, prognosis, and optimal selection of therapy.”
BackgroundExplainable COVID-19 Infections Identification and Delineation Using Calibrated Pseudo Labels
Li, M., Fang, Y., Tang, Z., Onuorah, C., Xia, J., Ser, J. D., Walsh, S. L. F., & Yang, G. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3189054 · Cited by 22
Self-citationArtificial intelligence applications used in the clinical response to COVID-19: A scoping review
Mann, S., Berdahl, C. T., Baker, L., & Girosi, F. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000132 · Cited by 24
Why an overreliance on AI-driven modelling is bad for science
Narayanan, A., & Kapoor, S. (2025) · Nature · doi:10.1038/d41586-025-01067-2 · Cited by 21
Lorde, N., Mahapatra, S., & Kalaria, T. (2024) · Diagnostics · doi:10.3390/diagnostics14161808 · Cited by 21
“…X-rays and CT scans, which relied on a publicly available dataset of paediatric radiographs as negative comparators, led to models that became efficient at differentiating adult from paediatric images, instead of differentiating COVID-19 patients from those with other causes of pneumonia [48,49].”
BackgroundCommon Pitfalls and Recommendations for Grand Challenges in Medical Artificial Intelligence
Reinke, A., Tizabi, M. D., Eisenmann, M., & Maier‐Hein, L. (2021) · European Urology Focus · doi:10.1016/j.euf.2021.05.008 · Cited by 22
Aslam, T. M., & Hoyle, D. C. (2021) · Ophthalmology and Therapy · doi:10.1007/s40123-021-00430-6 · Cited by 20
“as promising as they appear and must be subjected to further assessment to ensure that they are based on research conducted under strict methodological guidelines [3]; for example, a recent study highlights that impressive-appearing outcomes in some scientific reports are significantly flawed when subjected to close s…”
BackgroundMulrenan, C., Rhode, K., & Fischer, B. M. (2022) · Diagnostics · doi:10.3390/diagnostics12040869 · Cited by 21
“This review highlights a number of biases present in the literature, e.g., small sample size, potential for image duplication, differing image quality, extreme cases included in dataset, as also discussed by Roberts [37], and these biases limit the ability to translate AI into clinical practice.”
BackgroundInfluentialArtificial Intelligence in Lung Imaging
Choe, J., Lee, S. M., Hwang, H. J., Lee, S. M., Yun, J., Kim, N., & Seo, J. B. (2022) · Seminars in Respiratory and Critical Care Medicine · doi:10.1055/s-0042-1755571 · Cited by 21
Kim, T., Goh, T. S., Lee, J. S., Lee, J. H., Kim, H., & Jung, I. D. (2023) · Physical and Engineering Sciences in Medicine · doi:10.1007/s13246-023-01215-w · Cited by 21
Yearley, A. G., Goedmakers, C. M. W., Panahi, A., Doucette, J. A., Rana, A., Ranganathan, K., & Smith, T. R. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102607 · Cited by 21
Digital health and care: emerging from pandemic times
Peek, N. B., Sujan, M. A., & Scott, P. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100861 · Cited by 30
“20 Hence, we need to be cautious with the claims being made.”
BackgroundSelf-reported checklists and quality scoring tools in radiomics: a meta-research
Koçak, B., D’Antonoli, T. A., Kuş, E. A., Keleş, A., Kala, A., Köse, F., Kadıoğlu, M., Solak, S., Sunman, Ş., & Temiz, Z. H. (2024) · European Radiology · doi:10.1007/s00330-023-10487-5 · Cited by 21
“…including the complexity of the radiomic methodology with the vast number of configuration alternatives [6], disregarding potential modeling pitfalls [7], poor reporting [8], insufficient open science practices [9, 10], and a significant publication bias toward presenting positive results [11].”
López-Cabrera, J. D., Orozco‐Morales, R., Portal-Díaz, J. A., Enríquez, O. A. L., & Pérez-Díaz, M. (2021) · Health and Technology · doi:10.1007/s12553-021-00609-8 · Cited by 25
“In one review [94], it was determined that none of the articles analyzed in their research met the requirements to be considered reliable.”
BackgroundHaynes, S. C., Johnston, P., & Elyan, E. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18543-y · Cited by 23
A refined approach for evaluating small datasets via binary classification using machine learning
Steinert, S., Ruf, V., Dzsotjan, D., Großmann, N., Schmidt, A., Kühn, J., & Küchemann, S. (2024) · PLoS ONE · doi:10.1371/journal.pone.0301276 · Cited by 23
“This may lead to results that are not generalizable to other datasets and, therefore, are of limited use [4, 5].”
Machine learning techniques for CT imaging diagnosis of novel coronavirus pneumonia: a review
Chen, J., Li, Y., Guo, L., Zhou, X., Zhu, Y., He, Q., Han, H., & Feng, Q. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07709-0 · Cited by 21
“Therefore, a database with a systematic review of submitted data is more conducive to computer-aided diagnosis and treatment system than the immediate release of data of weak quality as a public database [104].”
Position statement on clinical evaluation of imaging AI
McCague, C., Mackay, K., Welsh, C., Constantinou, A., Jena, R., Crispin‐Ortuzar, M., McCague, C., Mackay, K., Welsh, C., Constantinou, A., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00090-0 · Cited by 20
Self-citationCOVID-19 Hierarchical Classification Using a Deep Learning Multi-Modal
Althenayan, A. S., Alsalamah, S., Aly, S. F., Nouh, T. A., Mahboub, B. H., Salameh, L. I. J., Alkubeyyer, M., & Mirza, A. A. (2024) · Sensors · doi:10.3390/s24082641 · Cited by 20
“It produces promising results for learning complex problems in radiology [12].”
BackgroundEnd-to-end reproducible AI pipelines in radiology using the cloud
Bontempi, D., Nürnberg, L., Pai, S., Krishnaswamy, D., Thiriveedhi, V., Hosny, A., Mak, R. H., Farahani, K., Kikinis, R., Fedorov, A., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-51202-2 · Cited by 23
Sooraksa, N. (2021) · International Conference on Engineering, Applied Sciences, and Technology (ICEAST) · doi:10.1109/iceast52143.2021.9426269 · Cited by 20
“For example, the disruption due to the COVID-19 pandemic has opened new research topics using tools in computational intelligence (CI) and artificial intelligence (AI) [1-6].”
BackgroundGalanty, M., Luitse, D. M. R., Noteboom, S. H., Croon, P. M., Vlaar, A. P. J., Poell, T., Sánchez, C. I., Blanke, T., & Išgum, I. (2024) · Scientific Reports · doi:10.1038/s41598-024-83218-5 · Cited by 23
Differential evolution and particle swarm optimization against COVID-19
Piotrowski, A., & Piotrowska, A. E. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-10052-w · Cited by 20
Utilizing artificial intelligence for environmental sustainability
Mumtaz, N., Izhar, T., Pandey, G., & Labhasetwar, P. (2022) · Elsevier eBooks · doi:10.1016/b978-0-323-90396-7.00010-9 · Cited by 20
Clinical Promise of Brain-Phenotype Modeling
Greene, A. S., & Constable, R. T. (2023) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2023.1419 · Cited by 20
FetMRQC: A robust quality control system for multi-centric fetal brain MRI
Sanchez, T., Estéban, O., Gomez, Y., Pron, A., Koob, M., Dunet, V., Girard, N. J., András, J., Eixarch, E., Auzias, G., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103282 · Cited by 22
“These observations were made possible by following good practices regarding evaluation and reporting of dataset with domain shifts (Roberts et al., 2021; Varoquaux and Cheplygina, 2022; Zech et al., 2018).”
Two-Strain COVID-19 Model Using Delayed Dynamic System and Big Data
Martsenyuk, V., Bernaś, M., & Kłos-Witkowska, A. (2021) · IEEE Access · doi:10.1109/access.2021.3104519 · Cited by 18
Shah, I., Doshi, C., Patel, M. M., Tanwar, S., Hong, W., & Sharma, R. (2022) · Medicina · doi:10.3390/medicina58020311 · Cited by 18
“Transfer learning is used to obtain advantages but over-parameterisation of models hinders performance, so there is a requirement for more accurate image processing techniques [145].”
BackgroundInfluentialSelf-citationChamberlin, J., Aquino, G. J., Schoepf, U. J., Nance, S., Godoy, F., Carson, L., Giovagnoli, V. M., Gill, C. E., McGill, L. J., O’Doherty, J., et al. (2022) · Academic Radiology · doi:10.1016/j.acra.2022.03.023 · Cited by 18
“(11) For each lobe, the disease extent was judged to be one of the following categories: (0) the lobe is not affected; (1) 1%-25%; (2) 25%-50%; (3) 50%-75%; and (4) 75%-100%.”
InfluentialPalmisano, A., Vignale, D., Boccia, E., Nonis, A., Gnasso, C., Leone, R., Montagna, M., Nicoletti, V., Bianchi, A. G., Brusamolino, S., et al. (2022) · La radiologia medica · doi:10.1007/s11547-022-01518-0 · Cited by 18
“Differently from most of previous predictive models [3, 4, 6, 30], in our study the outcome was defined as patients survival, considered as the most reliable endopoint during COVID-19 pandemic, due to scarce reliability of other endpoints affected by local protocols and hospital resources.”
BackgroundResultInfluentialPhysics-informed neural entangled-ladder network for inhalation impedance of the respiratory system
Kumar, A., Jain, S., Jain, S., Ritam, M., Xia, Y., & Chandra, R. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107421 · Cited by 22
Using alignment-free and pattern mining methods for SARS-CoV-2 genome analysis
Nawaz, M. S., Fournier‐Viger, P., Aslam, M., Li, W., He, Y., & Niu, X. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04618-0 · Cited by 18
“Roberts et al. [34] reviewed 62 published articles for COVID-19 diagnosis or prognosis from CXR or CT images.”
Liu, X., Reigle, J. K., Prasath, V. B. S., & Dhaliwal, J. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108093 · Cited by 18
Namdari, H., Moradikia, M., Zekavat, S. A. R., Askari, R., Mangoubi, O., & Petkie, D. T. (2024) · IEEE Transactions on AgriFood Electronics · doi:10.1109/tafe.2024.3455238 · Cited by 18
“Training ML models using raw data over a set of extracted features results in more accurate and robust training and testing results [43].”
Deep learning based image classification for embedded devices: A systematic review
Moreira, L. F. R., Moreira, R., Travençolo, B. A. N., & Backes, A. R. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129402 · Cited by 18
Integrating artificial intelligence in bedside care for covid-19 and future pandemics
Yu, M. A., Tang, A., Brown, K., Bouchakri, R., St-Onge, P., Wu, S., Reeder, J. C., Mullie, L., & Chassé, M. (2021) · BMJ · doi:10.1136/bmj-2021-068197 · Cited by 17
Comparison and ensemble of 2D and 3D approaches for COVID-19 detection in CT images
Ahmed, S. A. A., Yavuz, M. C., Şen, M. U., Gülşen, F., Tutar, O., Korkmazer, B., Samancı, C., Şirolu, S., Hamid, R., Eryurekli, A. E., et al. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.02.018 · Cited by 17
“In another recent and well-publicized survey, Roberts et al. [35] analyze all COVID-19 AI papers published in the first 9-months period of 2020, in terms of their potential potential biases, according to the criteria indicated in [36].”
Bortesi, J. P. T., Ranisau, J., Di, S., McGillion, M. H., Rosella, L. C., Johnson, A. E. W., Devereaux, P. J., & Petch, J. (2023) · Journal of Medical Internet Research · doi:10.2196/52880 · Cited by 19
“As has been noted in other reviews [33,41-43], it is difficult to assess RoB in the predictors of deep learning (DL) models that use images for prediction, as the image features are automatically selected by the algorithm.”
Detection of COVID-19 in Chest X-Ray Images Using a CNN Model toward Medical Applications
Mohsen, S., Scholz, S. G., & Elkaseer, A. (2024) · Wireless Personal Communications · doi:10.1007/s11277-024-11309-7 · Cited by 18
Turrisi, R., Verri, A., & Barla, A. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1360095 · Cited by 28
Wang, J., Li, J., Li, J., Yuan, C., Zhang, L., Jin, C., Xu, J., Wen, Y., Wen, Y., Li, B., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00496-3 · Cited by 17
“However, treatment information was not considered in the model developed in most prior studies, and the models were only implemented as classi fi cation tasks 27,28 , e.g., severity-level classi fi cation.”
MethodSivanesan, U., Wu, K., McInnes, M. D. F., Dhindsa, K., Salehi, F., & van der Pol, C. B. (2023) · Canadian Association of Radiologists Journal · doi:10.1177/08465371221134056 · Cited by 19
“14,15”
Domingo, J., Galal, G., Huang, J., Soni, P., Mukhin, V., Altman, C., Bayer, T., Byrd, T. F., Caron, S., Creamer, P., et al. (2022) · NEJM Catalyst · doi:10.1056/cat.21.0469 · Cited by 16
Metadata
- Authors
- Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., Weir-McCall, J. R., Teng, Z., Gkrania-Klotsas, E., AIX-COVNET, Ruggiero, A., Korhonen, A., Jefferson, E., Ako, E., Langs, G., Gozaliasl, G., Yang, G., Prosch, H., Preller, J., Stanczuk, J., Tang, J., Hofmanninger, J., Babar, J., Sánchez, L. E., Thillai, M., Gonzalez, P. M., Teare, P., Zhu, X., Patel, M., Cafolla, C., Azadbakht, H., Jacob, J., Lowe, J., Zhang, K., Bradley, K., Wassin, M., Holzer, M., Ji, K., Ortet, M. D., Ai, T., Walton, N., Lio, P., Stranks, S., Shadbahr, T., Lin, W., Zha, Y., Niu, Z., Rudd, J. H. F., Sala, E., & Schönlieb, C.-B.
- Affiliations
- AstraZeneca (United Kingdom) (GB)
- University of Cambridge (GB)
- University of Manchester (GB)
- Royal Papworth Hospital (GB)
- University of Dundee (GB)
- Royal Brompton Hospital (GB)
- Medical University of Vienna (AT)
- University of Helsinki (FI)
- Imperial College London (GB)
- Boehringer Ingelheim (Germany) (DE)
- Addenbrooke's Hospital (GB)
- University College London (GB)
- Macau University of Science and Technology (MO)
- Tongji Hospital (CN)
- Renmin Hospital of Wuhan University (CN)
- Corresponding author
- Roberts, Michael
- ORCID
- Roberts, Michael
- Thorpe, Matthew
- Gilbey, Julian
- Yeung, Michael
- Ursprung, Stephan
- McCague, Cathal
- Weir-McCall, Jonathan R.
- Gkrania-Klotsas, Effrossyni
- Ako, Emmanuel
- Langs, Georg
- Gozaliasl, Ghassem
- Yang, Guang
- Prosch, Helmut
- Preller, Jacobus
- Azadbakht, Hojjat
- Jacob, Joseph
- Ji, Kangyu
- Walton, Nicholas
- Stranks, Samuel
- Shadbahr, Tolou
- Rudd, James H. F.
- Sala, Evis
- Schönlieb, Carola-Bibiane
- Venue
- Nature Machine Intelligence
- Published
- 2021-03-15
- Volume
- 3
- Issue
- 3
- Pages
- 199-217
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W3119527628
- Cited by
- 1,044
- References
- 83
- Citations by year
- 2026: 203 · 2025: 96 · 2024: 143 · 2023: 248 · 2022: 224 · 2021: 130
- FWCI
- 98.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Machine Intelligence (publishedVersion, cc-by)
- Discovery Research Portal (University of Dundee) (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- elib (German Aerospace Center) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Työväentutkimus Vuosikirja (submittedVersion)
- arXiv (Cornell University)
- Funders
- AstraZeneca
- European Commission (777826)
- Medical Research Council (1966157, MR/S010351/1)
- Engineering and Physical Sciences Research Council (EP/T017961/1, EP/N014588/1, EP/S026045/1)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- COVID-19 pandemic, machine learning, chest radiograph, chest CT, COVID-19 detection, outcome prediction, clinical utility, model validation, methodological errors, bias, research recommendations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
