The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.
Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
Open accessAbstract
Abstract Machine learning methods offer great promise for fast and accurate detection and prognostication of coronavirus disease 2019 (COVID-19) from standard-of-care chest radiographs (CXR) and chest computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we consider all published papers and preprints, for the period from 1 January 2020 to 3 October 2020, which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. All manuscripts uploaded to bioRxiv, medRxiv and arXiv along with all entries in EMBASE and MEDLINE in this timeframe are considered. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher-quality model development and well-documented manuscripts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Much of the current evidence is based on retrospective analyses with limited prospective validation [19,28].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The success of deep learning in computer vision (CV) analysis has significantly advanced its clinical applications in comprehending biomedical images, including CT scans [88], positron emission tomography (PET)-CT scans [89], whole-slide images (WSIs) [90], and MRI scans [91].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [88]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Based on this study, we suggest that such a gold rush may give rise to un-engaged 'augmentation' which is highly risky from a learning and knowledge perspective for experts, AI companies, and consumers.”
From To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis (2022) · cites this paper as Roberts et al. 2021
In the citing paper
“Despite problems like the one discovered by Roberts et al. [195], the fact that ML systems process and analyze data differently from human beings can be a great benefit to science and society because these systems may be able to identify useful and innovative structures, properties, patterns, and relationships that human beings would not recognize.”
From The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool (2025) · cites this paper as [195]
In the citing paper
“Differences in image acquisition and processing may result in “shortcuts” where models take cues from non-clinical features.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“For example, COVID-19 prediction models have been demonstrated to learn which institution the images were obtained from rather than features pertaining to the underlying disease. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“The methodological flaws likely is a reflection of the lack of diversity in the development team to allow clinicians, data analysts and patient advocacy groups to work side-by-side throughout the AI life cycle. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“A similar result is summarized in a systematic review of COVID-19 prediction models on CT scans and Chest X-rays (CXRs) that found limited clinical utility due to methodological flaws and underlying biases stemming from small training data sets, data set variability, and limited integration of non-imaging data, among others.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Wang, D., Hu, B., Hu, C., Zhu, F., Liu, X., Zhang, J., Wang, B., Xiang, H., Cheng, Z., Xiong, Y., et al. (2020) · JAMA · doi:10.1001/jama.2020.1585 · Cited by 20,148
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Ai, T., Yang, Z., Hou, H., Zhan, C., Chen, C., Lv, W., Tao, Q., Sun, Z., & Xia, L. (2020) · Radiology · doi:10.1148/radiol.2020200642 · Cited by 5,793
Self-citationRadiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688
Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning
Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C. S., Liang, H., Baxter, S. L., McKeown, A., Yang, G., Wu, X., Yan, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.010 · Cited by 4,842
Self-citation
Research works printed but not linked: 7
- Wong, H. Y. F. et al. Frequency and distribution of chest radiographic findings in COVID-19 positive patients. Radiology 296, 201160 (2019).
- Bassi, P. R. A. S. & Attux, R. A deep convolutional neural network for COVID-19 detection using chest X-rays. Preprint at http://arxiv.org/abs/2005.01578 (2020). · arXiv
- Amyar, A., Modzelewski, R. & Ruan, S. Multi-task deep learning based CT imaging analysis for COVID-19: classification and segmentation. Preprint at medRxiv https://doi.org/10.1101/2020.04.16.2006470 (2020). · DOI
- Goodfellow, I. J. et al. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems Vol. 2 2672–2680 (MIT Press, 2014).
- Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations ICLR 2017 (ICLR, 2017).
- Covidence Systematic Review Software v2486 6284093b (Veritas Health Innovation, 2020); https://www.covidence.org/
- Aviles-Rivero, A. I., Sellars, P., Schönlieb, C.-B. & Papadakis, N. GraphXCOVID: explainable deep graph diffusion pseudo-labelling for identifying COVID-19 on chest X-rays. Preprint at https://arxiv.org/abs/2010.00378 (2020). · arXiv
Cited by
Sorted by citations
Items 501–600 of 1,000
Fonollosa, L. G., Webber, T., Brotons, J. M., Cerdà, M., Gillespie, D. M., Pirotta, E., & Rendell, L. E. (2024) · The Journal of the Acoustical Society of America · doi:10.1121/10.0034602 · Cited by 7
Qi, J., Li, M., Liu, J., Shu, Y., Yu, D. S. F., Ma, S., Cui, W., Zhao, Y., Chen, Y., Jiang, R., et al. (2026) · Academia AI and Applications · doi:10.20935/acadai8260 · Cited by 20
“During COVID-19, many imaging/triage models suffered from data quality issues and demographic bias, emphasizing perceive-stage noise, OOD generalization failures, and the need for stratified fairness audits (Section 3.1) [180, 181].”
InfluentialDifferent Algorithms (Might) Uncover Different Patterns: A Brain-Age Prediction Case Study
Ettling, T. J., Saba-Sadiya, S., & Roig, G. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385662 · Cited by 5
“First, we grouped features based on in which of the five frequency bands they originated (Fig.”
MethodTen quick tips for ensuring machine learning model validity
Goh, W. W. B., Kabir, M. N., Yoo, S., & Wong, L. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012402 · Cited by 5
“Current literature provides general guidelines on using ML models in different areas like chemical science, COVID-19 data, etc. [13–16], which focuses more on input data, leakage, reproducibility, class imbalance, parameter tuning, and choosing an appropriate metric.”
MethodNagaraj, Y., de Jonge, G., Андрейченко, А. Е., Presti, G., Fink, M. A., Pavlov, N. A., Quattrocchi, C. C., Morozov, S. P., Veldhuis, R. N. J., Oudkerk, M., et al. (2022) · European Radiology · doi:10.1007/s00330-022-08730-6 · Cited by 5
“In this study, (A) we successfully evaluated multi-site or public–private dataset scenarios and showed overall CO-RADS classification improvement by using noise reduction technique; (B) by choosing high-quality data from two clinical centers and heterogenous imaging protocols, we significantly reduced the population s…”
BackgroundMethodSurvival Analysis of COVID-19 Patients With Symptoms Information by Machine Learning Algorithms
Kim, G., Yoo, C. D., & Yang, S. J. (2022) · IEEE Access · doi:10.1109/access.2022.3182350 · Cited by 5
“There are five recommendations, given by [28], for highquality research on modeling COVID-19 data.”
BackgroundMethodGearhart, A. S., Dwork, N., & Jone, P. (2022) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2022.100082 · Cited by 7
A new model for classification of medical CT images using CNN: a COVID-19 case study
de Sousa, P. M., Carneiro, P. C., Pereira, G. M., Oliveira, M. M., da Costa, C. A., de Moura, L. V., Mattjie, C., da Silva, A. M. M., Macedo, T. A. A., & Patrocínio, A. C. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-14316-7 · Cited by 5
“So that the model’s performance results could be safely compared, we created criteria for exclusion of the images based in [39], with the help of a radiologist.”
BackgroundInfluentialMetamorphic Testing of Classification Program for the COVID-19 Intelligent Diagnosis
Ma, Y., Pan, Y., & Fan, Y. (2022) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa56465.2022.00031 · Cited by 5
“Six model evaluation metrics [9], which are commonly used in machine learning classification programs of medical diagnosis and introduced to determine the consistency of classification results.”
MethodDigital Case Manager—A Data-Driven Tool to Support Family Caregivers with Initial Guidance
Wunderlich, P., Wiegräbe, F., & Dörksen, H. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20021215 · Cited by 5
“This is particularly true for diagnostic methods that provide efficient data for model learning, such as image-based procedures like magnetic resonance imaging (MRI) [11], computer tomography (CT) [12,13], positron emission tomography (PET) [14], etc.”
Wang, H., Meng, X., Tang, Q., Ye, H., Luo, Y., & Li, J. (2023) · Journal of Healthcare Engineering · doi:10.1155/2023/7139560 · Cited by 5
“Stakeholders show rising concern on the quality of the AIMD, such as its comparability [20] and transparency [21], which poses considerable challenges to regulation compared to a conventional medical device.”
BackgroundOlar, A., Biricz, A., Bedőházi, Z., Sulyok, B., Pollner, P., & Csabai, I. (2023) · Scientific Reports · doi:10.1038/s41598-023-30505-2 · Cited by 5
“Studies have compared and evaluated popular CNNs 17,18 for COVID-19 CXRs through accuracy metrics and input manipulation, some incorporating multiple datasets 4,19 and highlighting common pitfalls 20 .”
MethodAUTHENTICITY AND THE POOR IMAGE IN THE AGE OF DEEP LEARNING
Wasielewski, A. (2023) · Photographies · doi:10.1080/17540763.2023.2189158 · Cited by 5
Li, Z., Xing, Q., Zhao, J., Yu, M., Zhang, K., Feng, G., Qu, F., Li, Y., He, W., Shi, W., et al. (2023) · IEEE Access · doi:10.1109/access.2023.3279402 · Cited by 5
“Inspired by the ideas [40], [41], to prevent over-fitting and further verify the efficiency of the proposed model, another CXR image datasets with different features is used in this paper, termed as the Dataset-2.”
MethodSelf-citationAgarwal, S., Wood, D. A., Marc Modat, A., & Booth, T. C. (2023) · Journal of NeuroInterventional Surgery · doi:10.1136/jnis-2023-020218 · Cited by 5
“14 In this study, the authors have made no distinction between studies that have validated internally or externally, neither in the summary table nor in the QUADAS-2 risk of bias assessment.”
BackgroundCOVID-19 and the Data Governance Gap
Austin, L. M. (2023) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-050520-101947 · Cited by 5
Considerations in the assessment of machine learning algorithm performance for medical imaging
Gossmann, A., Sahiner, B., Samala, R. K., Wen, S., Cha, K. H., & Petrick, N. (2024) · Elsevier eBooks · doi:10.1016/b978-0-32-385124-4.00029-5 · Cited by 5
Okada, N., Umemura, Y., Shi, S., Inoue, S., Honda, S., Matsuzawa, Y., Hirano, Y., Kikuyama, A., Yamakawa, M. Y., Gyobu, T., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-52135-y · Cited by 7
“For example, in steps one, two, and six, most previous studies still need to present that their test datasets comprehensively include diseases that should be differentiated from COVID-19 15 .”
BackgroundPatil, P., & Narawade, V. E. (2024) · International Research Journal of Multidisciplinary Technovation · doi:10.54392/irjmt24211 · Cited by 5
Zandehshahvar, M., van Assen, M., Kim, E. K., Kiarashi, Y., Keerthipati, V., Tessarin, G., Muscogiuri, E., Stillman, A. E., Filev, P. D., Davarpanah, A. H., et al. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01151-5 · Cited by 5
“Despite the development of numerous AI algorithms, their clinical implementation has been hindered by the lack of transparency and reliability in their predictions [24, 25].”
AI-Assisted CT as a Clinical and Research Tool for COVID-19
Tse, Z. T. H., Hovet, S., Ren, H., Barrett, T., Xu, S., Türkbey, B., & Wood, B. J. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.590189 · Cited by 5
“To increase speci fi city and reduce the analytical burden on radiologists, AI could be utilized to learn the unique characteristics of chest CT in COVID-19 compared to other diseases (Roberts et al., 2021).”
BackgroundValidation pipeline for machine learning algorithm assessment for multiple vendors
Bizzo, B. C., Ebrahimian, S., Walters, M. E., Michalski, M. H., Andriole, K. P., Dreyer, K. J., Kalra, M. K., Alkasab, T. K., & Digumarthy, S. R. (2022) · PLoS ONE · doi:10.1371/journal.pone.0267213 · Cited by 5
“Several publications have raised concern over lack of generalizability and robustness data for both the FDA-cleared as well as research AI algorithms [16].”
BackgroundAlNuaimi, D., & AlKetbi, R. (2022) · BJR|Open · doi:10.1259/bjro.20210075 · Cited by 5
Statsenko, Y. A., Habuza, T., Talako, T., Kurbatova, T., Simiyu, G. L., Smetanina, D., Sido, J., Qandil, D. S., Meribout, S., Gelovani, J. G., et al. (2022) · IEEE Access · doi:10.1109/access.2022.3211080 · Cited by 5
“Finally , applying artificial intelligence to the consecutive studies of the laboratory findings and medical imaging dwells promises to combine the evidence coming from the diagnostic modalities based on conceptually different methodologies [34], [35], [36].”
BackgroundMethodBaral, B., Muduli, K., Jakhmola, S., Indari, O., Jangir, J., Rashid, A. H., Jain, S., Mohapatra, A. K., Patro, S., Parida, P., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3263431 · Cited by 5
“Studies have reported using the deep learning (DL) approach for detecting COVID-19 from chest CT [25], [26], [27].”
MethodAutomatic COVID-19 and Common-Acquired Pneumonia Diagnosis Using Chest CT Scans
Motta, P. C., Cortez, P. C., Silva, B. R. D. S., Yang, G., & de Albuquerque, V. H. C. (2023) · Bioengineering · doi:10.3390/bioengineering10050529 · Cited by 5
“While deep learning models have shown promise in both diagnosis and prognosis tasks, it is essential to recognize the limitations of these models and use them in conjunction with other clinical information and expertise [28,29].”
BackgroundSelf-citationBuchsbaum, J. C., Espey, M. G., Obcemea, C., Capala, J., Ahmed, M. M., Prasanna, P. G., Vikram, B., Hong, J. A., Teicher, B., Aryankalayil, M. J., et al. (2022) · Journal of Clinical Oncology · doi:10.1200/jco.21.02579 · Cited by 5
Al‐Yousif, N., Komanduri, S., QURASHI, H. M. S., Korzhuk, A., Lawal, H. O., Abourizk, N., Schaefer, C., Mitchell, K. J., Dietz, C. M., Hughes, E. K., et al. (2022) · medRxiv · doi:10.1101/2022.06.10.22276249 · Cited by 5
Artificial intelligence, machine learning, and reproducibility in stroke research
Romoli, M., & Caliandro, P. (2024) · European Stroke Journal · doi:10.1177/23969873241275863 · Cited by 5
Recent advances and future prospects for blockchain in biomedicine
Ni, E., Tang, X., Zhou, X., Lee, D. H., Elhussein, A., Knight, E., Gürsoy, G., & Gerstein, M. (2025) · Cell Reports Methods · doi:10.1016/j.crmeth.2025.101114 · Cited by 4
“63,64 This system of trust and transparency enabled by blockchain can facilitate the integration of ML in distributed and collaborative health systems and biomedical research settings, accelerating the pace of innovation in the field.”
Development and Validation of Echocardiography Artificial Intelligence Models: A Narrative Review
Bennett, S., Johnson, C. L., Fisher, G., Erskine, F., Krasner, S., Fletcher, A. J., & Leeson, P. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14197066 · Cited by 5
“While many AI models demonstrate high performance in retrospective and prospective studies, these findings often rely on curated datasets and may not be generalisable to the wider patient populations or clinical workflows [58,59].”
Avellanet, M., Sánchez‐Raya, J., Maccarone, M. C., & Dionyssiotis, Y. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1774697 · Cited by 4
“Much research relied on retrospective, single-center data, frequently characterized by small sample numbers, which heightens the danger of overestimating model efficacy and constrains real-world clinical relevance (14).”
Dash, S., Patil, K. R., Bali, A., Raskar, I. R., Dongre, Y. V., Bhosle, A. A., & Meshram, V. A. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1787508 · Cited by 4
Physics-inspired explainable AI for mechanistic and decision support in drug discovery
Lavecchia, A. (2026) · Drug Discovery Today · doi:10.1016/j.drudis.2026.104679 · Cited by 4
Towards Reliable Data Augmentation in Machine Learning: Practices to Prevent Data Leakage
Gerz, F., & Jelali, M. (2026) · Smart Agricultural Technology · doi:10.1016/j.atech.2026.102462 · Cited by 5
Modeling Human Innate Immune Response Using Graph Neural Networks
Henna, S. (2021) · IEEE Access · doi:10.1109/access.2021.3133809 · Cited by 4
Linking Symptom Inventories Using Semantic Textual Similarity
Kennedy, E., Vadlamani, S., Lindsey, H. M., Peterson, K., Dams-O’Connor, K., Agarwal, R., Amiri, H., Andersen, R. K., Babikian, T., Baron, D. A., et al. (2025) · Journal of Neurotrauma · doi:10.1089/neu.2024.0301 · Cited by 5
“Following AI safety and reporting requirements, 30,31 we tested this hypothesis by assessing the ability of semantically linked items to estimate cross-inventory symptom scores for thousands of patients with TBI who were dually assessed on different inventories.”
InfluentialImproving the diagnostic accuracy using amplification and sequencing of the SARS-CoV-2 genome
Kumari, N. M. J., Bhattacharyya, D., & Rao, N. T. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-821318-6.00002-5 · Cited by 4
C-Libras: A Gesture Recognition App for the Brazilian Sign Language
Trotta, T., da Rocha, L. C. D., de Andrade, T. R., Guimarães, M. D. P., & Dias, D. R. C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-10522-7_41 · Cited by 4
Artificial intelligence in practice
Rangarajan, K. (2022) · Cancer Research Statistics and Treatment · doi:10.4103/crst.crst_128_22 · Cited by 4
Muiños, P. J. A., Vicente-Palacios, V., Pérez-Sánchez, P., Sampedro-Gómez, J., Sánchez-Puente, A., Dorado-Díaz, P. I., y, L. N.-F., Salinas, P., Gutiérrez-García, H., Amat-Santos, I. J., et al. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm12091413 · Cited by 4
“An example is the multitude of publications on the prediction of COVID-19 in medical imaging that have appeared in recent years and do not yield clear results or predictors [29].”
BackgroundSantos, M. A. D. C., & Berton, L. (2023) · IEEE Access · doi:10.1109/access.2023.3325404 · Cited by 4
“In [9] the authors present a review of the works published from January 1, 2020, to October 3, 2020, analyzing the proposed approaches, sets of images used, and pre-processing techniques, among other factors.”
BackgroundDiscovery Viewer (DV): Web-Based Medical AI Model Development Platform and Deployment Hub
Fauveau, V., Sun, S., Liu, Z., Mei, X., Grant, J., Sullivan, M., Greenspan, H. K., Feng, L., & Fayad, Z. A. (2023) · Bioengineering · doi:10.3390/bioengineering10121396 · Cited by 4
“The study found that none of these models were really suitable for detecting or diagnosing COVID-19 from standard medical imaging [2].”
BackgroundSeeing the Whole Elephant - A Comprehensive Framework for Data Education
Cruickshank, I. J., Bastian, N. D., Blair, J. R. S., Chewar, C. M., & Sobiesk, E. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630922 · Cited by 4
Artificial Intelligence in Otolaryngology
Tai, K., Zhao, R., & Rameau, A. (2024) · Otolaryngologic Clinics of North America · doi:10.1016/j.otc.2024.04.008 · Cited by 4
Malik, S., & Rathee, P. (2024) · SN Computer Science · doi:10.1007/s42979-024-03103-w · Cited by 4
“By integrating XAI tech - niques, we aim to provide transparent and interpretable insights into the model’s decision-making process, enhanc - ing its accuracy and trustworthiness [ 5 ].”
Schenk, P. O., & Kern, C. A. (2024) · AStA Wirtschafts- und Sozialstatistisches Archiv · doi:10.1007/s11943-024-00344-2 · Cited by 11
Enhancing Medical Image Classification with Unified Model Agnostic Computation and Explainable AI
Neghawi, E., & Liu, Y. (2024) · AI · doi:10.3390/ai5040111 · Cited by 4
“Most available datasets are sourced from a limited number of healthcare institutions, often in high-income countries, leading to biases that affect the generalizability of models to broader or underserved populations [17,18].”
Agnihotri, A., & Kohli, N. (2025) · Recent Advances in Electrical & Electronic Engineering (Formerly Recent Patents on Electrical & Electronic Engineering) · doi:10.2174/0123520965334135241115064754 · Cited by 4
Learning How to MIMIC: Using Model Explanations to Guide Deep Learning Training
Watson, M., Hasan, B. A. S., & Moubayed, N. A. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00151 · Cited by 4
Zhang, S., Zou, B., Xu, B., Su, J., & Hu, H. (2021) · IEEE International Conference on Progress in Informatics and Computing · doi:10.1109/pic53636.2021.9687080 · Cited by 4
Kothari, R. P., Chiu, C., Moukheiber, M., Jehiro, M., Bishara, A. M., Lee, C., Pirracchio, R., & Celi, L. A. (2022) · Anaesthesia Critical Care & Pain Medicine · doi:10.1016/j.accpm.2022.101126 · Cited by 4
Mirniaharikandehei, S., Abdihamzehkolaei, A., Choquehuanca, A., López, M. A., Pacheco, W., Estacio, L., Cahui, V., Huallpa, L., Quiñonez, K., Calderón, V., et al. (2023) · Bioengineering · doi:10.3390/bioengineering10030321 · Cited by 4
“, ranging from 90–100% accuracy [15]), no previous DL model is robust and clinically acceptable due to training bias and a “black-box” type approach [16].”
BackgroundBashir, M., Inzamam, W., Banday, M. K., Rasool, S. R., Bhat, M. H., Vlădulescu, C., Al-Misned, F. A., & El‐Serehy, H. A. (2023) · Diagnostics · doi:10.3390/diagnostics13122091 · Cited by 4
“There is no doubt about the usefulness of chest HRCT in COVID-19 patients, however, the role of LUS in COVID-19 has not been fully validated [12].”
BackgroundHunter, E., Koutsothanasi, C., Wilson, A. J., Santos, F. C., Salter, M., Westra, J. W., Powell, R., Dring, A. M., Brajer, P., Egan, B., et al. (2021) · medRxiv · doi:10.1101/2021.06.21.21259145 · Cited by 4
COVID19 Diagnosis Using Chest X-rays and Transfer LearningPreprint
Stubblefield, J., Causey, J., Dale, D. S., Qualls, J., Bellis, E. S., Fowler, J., Walker, K., & Huang, X. (2022) · medRxiv · doi:10.1101/2022.10.09.22280877 · Cited by 4
“These deep-learning models performed well on their respective datasets with reported area-under the receiver operating characteristic curve (AUROC) between 0.7 and 1.0 [17].”
BackgroundMethodInfluentialCao, W., Pomeroy, M. J., Liang, Z., Gao, Y., Shi, Y., Tan, J., Han, F., Wang, J., Ma, J., Lu, H., et al. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01178-8 · Cited by 3
“The data limitation is currently a well-known challenge for ML-based CADx [48].”
Wilson, B., Natali, C., Roach, M., Scott, D., Rahat, A., Rawlinson, D. R., & Cabitza, F. (2025) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-025-09514-4 · Cited by 6
Prado, Á. M., Guerrero-López, A., Arias-Londoño, J. D., & Godino-Llorente, J. I. (2025) · IEEE Access · doi:10.1109/access.2025.3607282 · Cited by 3
SIMBA: A robust and generalizable measure of data imbalance
Pivin-Bachler, J. R., & van den Broek, E. L. (2025) · Patterns · doi:10.1016/j.patter.2025.101395 · Cited by 3
Raieli, S., Jeanray, N., Gérart, S., Vachenc, S., & Altahhan, A. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-025-00056-0 · Cited by 7
Olawade, D. B., Osborne, A., Soladoye, A. A., Oluwadare, O. E., Awogbindin, E., & Wada, O. Z. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106313 · Cited by 3
Yang, Y., & Peters, M. A. (2026) · International Journal of Chinese Education · doi:10.1177/2212585x251413788 · Cited by 5
“AI is framed as a strategic instrument for achieving collectively de fi ned social objectives, including balanced regional development, mass educational inclusion, and the cultivation of future-oriented skills aligned with national development strategies (Roberts et al., 2021).”
Faa, G., Lai, E., Cau, F., Coghe, F., Rugge, M., Suri, J. S., Codipietro, C., Congiu, B., Graziano, S., Tiwari, E., et al. (2026) · Cancers · doi:10.3390/cancers18071184 · Cited by 5
Coombs, L. P., Brink, L., Kim, W., Wald, C., Schmidt, K., Tilkin, M., Dreyer, K., Cook, T. S., & Bizzo, B. C. (2026) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2026.04.024 · Cited by 3
Vaidya, P., Alilou, M., Hiremath, A., Gupta, A. B., Bera, K., Furin, J. J., Armitage, K. B., Gilkeson, R., Yuan, L., Fu, P., et al. (2022) · Frontiers in Radiology · doi:10.3389/fradi.2022.781536 · Cited by 3
“In the recent study by Roberts et al. (40), the authors point out that many recent AI/machine learning studies on diagnosis and prognosis of COVID-19 from radiographic scans are not reproducible and would not be clinically deployable.”
Adaptation to CT Reconstruction Kernels by Enforcing Cross-Domain Feature Maps Consistency
Shimovolos, S., Shushko, A., Belyaev, M., & Shirokikh, B. (2022) · Journal of Imaging · doi:10.3390/jimaging8090234 · Cited by 3
“Finally, we note that there is no standardized benchmark for the COVID-19 segmentation task [27].”
BackgroundMachine Learning for Clinical Electrophysiology
Loewe, A., Luongo, G., & Sánchez, J. (2022) · Lecture notes in bioengineering · doi:10.1007/978-981-19-6649-1_6 · Cited by 4
Gusev, A. V., Vlаdzimirskiy, A. V., & Gavrilenko, G. G. (2022) · Medical Technologies Assessment and Choice · doi:10.17116/medtech20224403112 · Cited by 3
COVID-19 therapy optimization by AI-driven biomechanical simulations
Agrimi, E., Diko, A., Carlotti, D. N., Ciardiello, A., Borthakur, M. P., Giagu, S., Melchionna, S., & Voena, C. (2023) · The European Physical Journal Plus · doi:10.1140/epjp/s13360-023-03744-5 · Cited by 4
Hu, J., Mougiakakou, S. G., Xue, S., Afshar‐Oromieh, A., Hautz, W. E., Christe, A., Sznitman, R., Rominger, A., Ebner, L., & Shi, K. (2023) · The European Physical Journal Plus · doi:10.1140/epjp/s13360-023-03745-4 · Cited by 3
“Although AI has demonstrated potential value in COVID diagnosis, pitfalls of AI have been reported, and the developed AI methods lack potential clinical use due to methodological flaws and/or underlying biases [54].”
BackgroundZhang, R., Griner, D., Garrett, J. W., Qi, Z., & Chen, G. (2023) · Scientific Reports · doi:10.1038/s41598-023-39855-3 · Cited by 4
SARS detection in chest CT scan images using the bootstrapped ViT-B/16 model
Appati, J. K., Ziamah, B., Akrofi, H. A., & Dodoo, A. A. (2025) · Iran Journal of Computer Science · doi:10.1007/s42044-025-00231-1 · Cited by 3
“Moreover, these features can overlap with other respiratory conditions, such as community-acquired pneumonia (CAP) and viral pneumonia, making differential diagnosis challenging [26].”
Blair, J. D., Khidas, K., & Marshall, K. E. (2025) · PLoS ONE · doi:10.1371/journal.pone.0329482 · Cited by 3
“While synthetic images produced from renders of 3D assets often appear photorealistic, computer vision models may still pick up on subtle differences, thus making model generalization across domains more difficult [33–36].”
Christakis, N., Tirchas, P., Politis, M., Achladianakis, M., Avgenikou, E., & Kossioris, G. (2021) · International Journal of Neural Networks and Advanced Applications · doi:10.46300/91016.2021.8.4 · Cited by 3
Predicting Severe Disease and Critical Illness on Initial Diagnosis of COVID-19: Simple Triage Tools
Kurban, L. A. S., Aldhaheri, S., Elkkari, A., Khashkhusha, R., Eissaee, S. A., Alzaabi, A. A., Ismail, M., & Bakoush, O. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.817549 · Cited by 3
“A recent review of over 400 diagnostic and prognostic AI triage tools for COVID 19 infection highlighted serious methodological and reporting flaws that jeopardize reproducibility, generalizability, and usability for clinical practice (48).”
BackgroundIskander, M. F., Leong, C., Shukla, P., Clemens, S., & Yun, Z. (2022) · IEEE Access · doi:10.1109/access.2022.3208143 · Cited by 3
Segall, R. S., & Sankarasubbu, V. (2022) · International Journal of Artificial Intelligence and Machine Learning · doi:10.4018/ijaiml.313574 · Cited by 3
Lasagni, G. (2022) · Legal studies in international, european and comparative criminal law · doi:10.1007/978-3-031-13952-9_8 · Cited by 3
Automated Detection of Covid-19 Waves with Computerized Tomography Scan Using Deep Learning
Suganya, D., & Kalpana, R. (2023) · Springer eBooks · doi:10.1007/978-3-031-13577-4_3 · Cited by 3
Casey, A., Ansari, S., Nakisa, B., Kelly, B., Brown, P., Cooper, P., Muhammad, I., Livingstone, S., Reddy, S., & Mäkinen, V. (2023) · JMIR AI · doi:10.2196/42313 · Cited by 5
“Yet, most AI applications in health care have not demonstrated improvement in clinical or health care outcomes [5,7].”
BackgroundCollective intelligent strategy for improved segmentation of COVID-19 from CT
Pal, S., Mitra, S., & Shankar, B. U. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121099 · Cited by 3
Spectral–temporal saliency masks and modulation tensorgrams for generalizable COVID-19 detection
Zhu, Y. M., & Falk, T. H. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101620 · Cited by 3
Tadesse, G. A., Ogallo, W., Cintas, C., Speakman, S., Walcott-Bryant, A., & Wayua, C. (2024) · npj Women s Health · doi:10.1038/s44294-024-00017-z · Cited by 3
“Thisisbestencapsulatedbytheadage garbage in, garbage out , meaning that if the quality of the data used for analysis is substandard, the insights derived from the analysis are likely to be questionable 13,14 .”
Artificial Intelligence Applications in Neonatal Critical Care: A Scoping Review
Sakore, S. S., Devi, S., Mahapure, P., Kamble, M., & Jadhav, P. (2024) · Journal of Clinical Neonatology · doi:10.4103/jcn.jcn_13_24 · Cited by 3
Min, X., Liu, Y., Zhou, S., Huang, H., Zhang, L., Gong, X., Yang, D., Wang, M., Yang, R., & Zhong, M. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01392-y · Cited by 3
Alumnado subrepresentado e inteligencia artificial
Casas, R. M. R., González, R. F., & Santos-Garrido, M. (2024) · European Public & Social Innovation Review · doi:10.31637/epsir-2025-843 · Cited by 6
A Pipeline for Automated Quality Control of Chest Radiographs
SELBY, I. A., González-Solares, E., Breger, A., Roberts, M. T., Sánchez, L. E., Babar, J. L., Rudd, J. H. F., Walton, N. A., Sala, E., Schönlieb, C., et al. (2025) · Radiology Artificial Intelligence · doi:10.1148/ryai.240003 · Cited by 3
Self-citationPrediction of the pneumonia from the CT lung images by using the multiband google NET CNN
Kumar, J. S., Pradeepa, R., Arulkarthick, & Chandragandhi, S. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.107738 · Cited by 3
PARADIM: A Platform to Support Research at the Interface of Data Science and Medical Imaging
Lemaréchal, Y., Couture, G., Pelletier, F., Lefol, R., Asselin, P.-L., Ouellet, S., Bernard, J., Ebrahimpour, L., Manem, V. S., Topalis, J., et al. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01554-y · Cited by 3
Rapid detection of SARS-CoV-2 using a radiolabeled antibody
Pirovano, G., Ordoñez, A. A., Jain, S. K., Reiner, T., Carroll, L. S., & Pillarsetty, N. (2021) · Nuclear Medicine and Biology · doi:10.1016/j.nucmedbio.2021.05.002 · Cited by 3
“they detect the presence of IgA, IgM, and IgG antibodies that are produced against the virus [16]; other examples of techniques include Raman imaging [17] and machine learning [18].”
Survey of Applications of Neural Networks and Machine Learning to COVID-19 Predictions
Segall, R. S. (2021) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-7998-8455-2.ch002 · Cited by 3
Carrero, R. S. D. V., Martínez‐Urbistondo, D., Cuevas‐Sierra, A., Ibañez-Sustacha, I., Candela-Fernandez, A., Dominguez-Calvo, A., Ramos-López, O., Vargas, J. A., Reglero, G., Villares-Fernández, P., et al. (2022) · Journal of Clinical Medicine · doi:10.3390/jcm11133578 · Cited by 3
Kataoka, Y., Baba, T., Ikenoue, T., Matsuoka, Y., Matsumoto, J., Kumasawa, J., Tochitani, K., Funakoshi, H., Hosoda, T., Kugimiya, A., et al. (2022) · Annals of Clinical Epidemiology · doi:10.37737/ace.22014 · Cited by 3
Majumder, D. (2022) · Journal of Medical Physics · doi:10.4103/jmp.jmp_26_22 · Cited by 3
“[24] Here, we have developed an algorithm within a free and open-source platform (software) and made simple statistical inference to distinguish between COVID-19, viral pneumonia (VP), and bacterial pneumonia (BP) cases.”
BackgroundMethodInfluentialVega, C., Kratochvíl, M., Satagopam, V., & Schneider, R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-07802-6_30 · Cited by 3
Shenouda, M., Flerlage, I., Kaveti, A., Giger, M. L., & Armato, S. G. (2023) · Journal of medical imaging · doi:10.1117/1.jmi.10.6.064504 · Cited by 2
“9 These biases may result in a lack of reproducibility and generalizability of the models developed, a common shortcoming recognized by the AI community.”
BackgroundResultRethinking diagnostics for pandemic readiness
(2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00883-5 · Cited by 3
Alami, H., Lehoux, P., Papoutsi, C., Shaw, S. E., Fleet, R. P., & Fortin, J. (2023) · Research Square · doi:10.21203/rs.3.rs-2546681/v1 · Cited by 3
AI for Medical Image Processing
Chepelev, L. L., Nicolaou, S., & Sheikh, A. (2023) · AI in Clinical Medicine · doi:10.1002/9781119790686.ch34 · Cited by 3
Metadata
- Authors
- Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., Weir-McCall, J. R., Teng, Z., Gkrania-Klotsas, E., AIX-COVNET, Ruggiero, A., Korhonen, A., Jefferson, E., Ako, E., Langs, G., Gozaliasl, G., Yang, G., Prosch, H., Preller, J., Stanczuk, J., Tang, J., Hofmanninger, J., Babar, J., Sánchez, L. E., Thillai, M., Gonzalez, P. M., Teare, P., Zhu, X., Patel, M., Cafolla, C., Azadbakht, H., Jacob, J., Lowe, J., Zhang, K., Bradley, K., Wassin, M., Holzer, M., Ji, K., Ortet, M. D., Ai, T., Walton, N., Lio, P., Stranks, S., Shadbahr, T., Lin, W., Zha, Y., Niu, Z., Rudd, J. H. F., Sala, E., & Schönlieb, C.-B.
- Affiliations
- AstraZeneca (United Kingdom) (GB)
- University of Cambridge (GB)
- University of Manchester (GB)
- Royal Papworth Hospital (GB)
- University of Dundee (GB)
- Royal Brompton Hospital (GB)
- Medical University of Vienna (AT)
- University of Helsinki (FI)
- Imperial College London (GB)
- Boehringer Ingelheim (Germany) (DE)
- Addenbrooke's Hospital (GB)
- University College London (GB)
- Macau University of Science and Technology (MO)
- Tongji Hospital (CN)
- Renmin Hospital of Wuhan University (CN)
- Corresponding author
- Roberts, Michael
- ORCID
- Roberts, Michael
- Thorpe, Matthew
- Gilbey, Julian
- Yeung, Michael
- Ursprung, Stephan
- McCague, Cathal
- Weir-McCall, Jonathan R.
- Gkrania-Klotsas, Effrossyni
- Ako, Emmanuel
- Langs, Georg
- Gozaliasl, Ghassem
- Yang, Guang
- Prosch, Helmut
- Preller, Jacobus
- Azadbakht, Hojjat
- Jacob, Joseph
- Ji, Kangyu
- Walton, Nicholas
- Stranks, Samuel
- Shadbahr, Tolou
- Rudd, James H. F.
- Sala, Evis
- Schönlieb, Carola-Bibiane
- Venue
- Nature Machine Intelligence
- Published
- 2021-03-15
- Volume
- 3
- Issue
- 3
- Pages
- 199-217
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W3119527628
- Cited by
- 1,044
- References
- 83
- Citations by year
- 2026: 203 · 2025: 96 · 2024: 143 · 2023: 248 · 2022: 224 · 2021: 130
- FWCI
- 98.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Machine Intelligence (publishedVersion, cc-by)
- Discovery Research Portal (University of Dundee) (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- elib (German Aerospace Center) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Työväentutkimus Vuosikirja (submittedVersion)
- arXiv (Cornell University)
- Funders
- AstraZeneca
- European Commission (777826)
- Medical Research Council (1966157, MR/S010351/1)
- Engineering and Physical Sciences Research Council (EP/T017961/1, EP/N014588/1, EP/S026045/1)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- COVID-19 pandemic, machine learning, chest radiograph, chest CT, COVID-19 detection, outcome prediction, clinical utility, model validation, methodological errors, bias, research recommendations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
