The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.
Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
Open accessAbstract
Abstract Machine learning methods offer great promise for fast and accurate detection and prognostication of coronavirus disease 2019 (COVID-19) from standard-of-care chest radiographs (CXR) and chest computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we consider all published papers and preprints, for the period from 1 January 2020 to 3 October 2020, which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. All manuscripts uploaded to bioRxiv, medRxiv and arXiv along with all entries in EMBASE and MEDLINE in this timeframe are considered. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher-quality model development and well-documented manuscripts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Much of the current evidence is based on retrospective analyses with limited prospective validation [19,28].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The success of deep learning in computer vision (CV) analysis has significantly advanced its clinical applications in comprehending biomedical images, including CT scans [88], positron emission tomography (PET)-CT scans [89], whole-slide images (WSIs) [90], and MRI scans [91].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [88]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Based on this study, we suggest that such a gold rush may give rise to un-engaged 'augmentation' which is highly risky from a learning and knowledge perspective for experts, AI companies, and consumers.”
From To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis (2022) · cites this paper as Roberts et al. 2021
In the citing paper
“Despite problems like the one discovered by Roberts et al. [195], the fact that ML systems process and analyze data differently from human beings can be a great benefit to science and society because these systems may be able to identify useful and innovative structures, properties, patterns, and relationships that human beings would not recognize.”
From The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool (2025) · cites this paper as [195]
In the citing paper
“Differences in image acquisition and processing may result in “shortcuts” where models take cues from non-clinical features.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“For example, COVID-19 prediction models have been demonstrated to learn which institution the images were obtained from rather than features pertaining to the underlying disease. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“The methodological flaws likely is a reflection of the lack of diversity in the development team to allow clinicians, data analysts and patient advocacy groups to work side-by-side throughout the AI life cycle. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“A similar result is summarized in a systematic review of COVID-19 prediction models on CT scans and Chest X-rays (CXRs) that found limited clinical utility due to methodological flaws and underlying biases stemming from small training data sets, data set variability, and limited integration of non-imaging data, among others.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Wang, D., Hu, B., Hu, C., Zhu, F., Liu, X., Zhang, J., Wang, B., Xiang, H., Cheng, Z., Xiong, Y., et al. (2020) · JAMA · doi:10.1001/jama.2020.1585 · Cited by 20,148
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Ai, T., Yang, Z., Hou, H., Zhan, C., Chen, C., Lv, W., Tao, Q., Sun, Z., & Xia, L. (2020) · Radiology · doi:10.1148/radiol.2020200642 · Cited by 5,793
Self-citationRadiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688
Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning
Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C. S., Liang, H., Baxter, S. L., McKeown, A., Yang, G., Wu, X., Yan, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.010 · Cited by 4,842
Self-citation
Research works printed but not linked: 7
- Wong, H. Y. F. et al. Frequency and distribution of chest radiographic findings in COVID-19 positive patients. Radiology 296, 201160 (2019).
- Bassi, P. R. A. S. & Attux, R. A deep convolutional neural network for COVID-19 detection using chest X-rays. Preprint at http://arxiv.org/abs/2005.01578 (2020). · arXiv
- Amyar, A., Modzelewski, R. & Ruan, S. Multi-task deep learning based CT imaging analysis for COVID-19: classification and segmentation. Preprint at medRxiv https://doi.org/10.1101/2020.04.16.2006470 (2020). · DOI
- Goodfellow, I. J. et al. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems Vol. 2 2672–2680 (MIT Press, 2014).
- Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations ICLR 2017 (ICLR, 2017).
- Covidence Systematic Review Software v2486 6284093b (Veritas Health Innovation, 2020); https://www.covidence.org/
- Aviles-Rivero, A. I., Sellars, P., Schönlieb, C.-B. & Papadakis, N. GraphXCOVID: explainable deep graph diffusion pseudo-labelling for identifying COVID-19 on chest X-rays. Preprint at https://arxiv.org/abs/2010.00378 (2020). · arXiv
Cited by
Sorted by citations
Items 401–500 of 1,000
Deep Learning Applied to Chest Radiograph Classification—A COVID-19 Pneumonia Experience
Furtado, A. N., Andrade, L., Frı́as, D., Maia, T., Badaró, R. J. D. S., & Nascimento, E. G. S. (2022) · Applied Sciences · doi:10.3390/app12083712 · Cited by 11
“The proposal of a new DL system based on the Inception V3 architecture, one that supports the identification of normal and abnormal CXR examinations and the diagnosis of COVID-19.”
MethodA deep learning-based application for COVID-19 diagnosis on CT: The Imaging COVID-19 AI initiative
Topff, L., Sánchez-García, J., López‐González, R., Jiménez-Pastor, A., Visser, J. J., Huisman, M., Guiot, J., Beets‐Tan, R. G. H., Alberich‐Bayarri, Á., Fuster-Matanzo, A., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0285121 · Cited by 15
Ballaz, S., Pulgar-Sánchez, M., Chamorro, K., & Fernández-Moreira, E. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101312 · Cited by 11
Shaban‐Nejad, A., Michalowski, M., Bianco, S., Brownstein, J. S., Buckeridge, D. L., & Davis, R. L. (2022) · Experimental Biology and Medicine · doi:10.1177/15353702221140406 · Cited by 10
Ethics of Artificial Intelligence in Medicine
Park, M., Ashwood, N., & Capes, N. (2025) · Cureus · doi:10.7759/cureus.83567 · Cited by 10
“If restricted data is used, it could lead to representation bias, reducing the efficiency and accuracy of AI in radiology [13].”
Lee, J. H., Ahn, J. S., Chung, M. J., Jeong, Y. J., Kim, J. H., Lim, J. K., Kim, J. Y., Kim, Y. J., Lee, J. E., & Kim, E. Y. (2022) · Sensors · doi:10.3390/s22135007 · Cited by 9
“It has been shown that previous imaging-based AI models for COVID-19 prognostication may be inadequate for real clinical usage [35,36].”
MethodA Clinician’s Guide to Understanding Bias in Critical Clinical Prediction Models
de Matos, J. C. R. G., Gallifant, J., Chowdhury, A., Economou-Zavlanos, N., Charpignon, M., Gichoya, J. W., Celi, L. A., Nazer, L. H., King, H., & Wong, A.-K. I. (2024) · Critical Care Clinics · doi:10.1016/j.ccc.2024.05.011 · Cited by 11
Zhou, H.-Y., Li, Y., Li, J., Meng, J., & Wu, A. (2024) · The Innovation Medicine · doi:10.59717/j.xinn-med.2024.100091 · Cited by 10
Gangwal, A., & Lavecchia, A. (2026) · Wiley Interdisciplinary Reviews Computational Molecular Science · doi:10.1002/wcms.70072 · Cited by 9
Monjur, O., Preo, R. B., Shams, A. B., Raihan, M. M. S., & Fairoz, F. (2021) · BioMed · doi:10.3390/biomed1020011 · Cited by 9
“Unfortunately, a recent extensive study demonstrated the lacking potential of this technique for clinical use, ascribing methodological flaws and/or underlying biases [8].”
BackgroundLi, X., Cui, L., Shonkwiler, S., & McMains, S. (2023) · Journal of Iron and Steel Research International · doi:10.1007/s42243-022-00907-z · Cited by 9
Catalano, M., Bortolotto, C., Nicora, G., Achilli, M. F., Consonni, A., Ruongo, L., Callea, G., Tito, A. L., Biasibetti, C., Donatelli, A., et al. (2023) · European Journal of Radiology Open · doi:10.1016/j.ejro.2023.100497 · Cited by 10
Local transplantation, adaptation, and creation of AI models for public health policy
Fournier-Tombs, É. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1085671 · Cited by 9
“Several pitfalls have been identified by researchers, including using the same, flawed datasets across many of these models, not properly selecting parameters, and failing to remove duplicate data (Roberts et al., 2021).”
BackgroundHybrid Quantum Machine Learning Assisted Classification of COVID-19 from Computed Tomography Scans
Sünkel, L., Martyniuk, D., Reichwald, J., Morariu, A., Seggoju, R. H., Altmann, P. J., Roch, C., & Paschke, A. (2023) · IEEE International Conference on Quantum Computing and Engineering (QCE) · doi:10.1109/qce57702.2023.00048 · Cited by 11
“Researchers around the world have therefore developed machine learning algorithms based on CT scans to properly detect and predict COVID-19 infection [25].”
BackgroundVega, C. (2021) · IEEE Access · doi:10.1109/access.2021.3095222 · Cited by 10
Bakker, L. J., Aarts, J. E. C. M., Groot, C. A. U., & Redekop, K. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01682-9 · Cited by 12
“have emphasised in their review of AI for the diagnosis and prognostication of secondary pneumonia, that many AI analytics were hampered by poor quality data [64].”
BackgroundZhu, Y. M., Mariakakis, A. T., de Lara, E., & Falk, T. H. (2022) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi56158.2022.9926950 · Cited by 9
“When combined with the findings reported above for Task 1, these findings suggest that the results obtained on the DI-COVA2 set are highly biased towards the training dataset, thus corroborating the findings reported for other input modalities (e.g., [8], [12], [13]).”
BackgroundResultMamandipoor, B., Bruno, R. R., Wernly, B., Wolff, G., Fjølner, J., Artigas, A., Pinto, B. B., Schefold, J. C., Kelm, M., Beil, M., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000136 · Cited by 10
“Indeed, a recent review on chest imaging, emphasised the importance of validation dataset to assess generalisability of the model to other cohorts, rather than only on the sampled population [16].”
BackgroundA few-shot approach for COVID-19 screening in standard and portable chest X-ray images
Cores, D., Vila-Blanco, N., Pérez-Alarcón, M., de Alegría, A. M., Mucientes, M., & Carreira, M. J. (2022) · Scientific Reports · doi:10.1038/s41598-022-25754-6 · Cited by 10
“At this moment there are very few high-quality datasets to build robust systems to detect COVID-19 based on CXR scan images 17 .”
BackgroundMethodUsing Artificial Intelligence for COVID-19 Detection in Blood Exams: A Comparative Analysis
Loddo, A., Meloni, G., & Pes, B. (2022) · IEEE Access · doi:10.1109/access.2022.3221750 · Cited by 9
“In particular, the automatic classification of COVID-19 has gained wide attention from researchers involved in the computer vision domain [23], [24], [25], [26], [27], [28], mainly thanks to the availability of imaging data like CXR or CT scans [16], [20], and even in low-end environments [29], [30].”
BackgroundLeveraging GANs for data scarcity of COVID-19: Beyond the hype
Ali, H., Grönlund, C., & Shah, Z. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00073 · Cited by 11
The use of gene expression datasets in feature selection research: 20 years of inherent bias?
Grisci, B. I., Feltes, B. C., Poloni, J. D. F., Narloch, P. H., & Dorn, M. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1523 · Cited by 8
“…learning models trained for the detection and prognostication of coro-navirus disease 2019 (COVID-19) from chest radiography or chest computed tomography images, Roberts et al. (2021) discovered that none could be used in clinical practice because of methodological flaws or underlying biases.”
MethodArtificial intelligence in respiratory pandemics—ready for disease X? A scoping review
Straub, J., Lobato, E. E., Páez, D., Langs, G., & Prosch, H. (2024) · European Radiology · doi:10.1007/s00330-024-11183-8 · Cited by 8
“Based on the fi ndings of previous studies [9, 12, 93], combined with the shortcomings identi fi ed in this work, we have compiled the following recommendations in Table 2.”
Self-citationLiu, B., Nie, Y., Song, H., Lin, Y., Li, J., Weng, X., Su, Z., Suo, Y., Lv, T., Zhao, X., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103557 · Cited by 10
Assessing feature scorer results on high-dimensional datasets with t-SNE
Grisci, B. I., Inostroza-Ponta, M., & Dorn, M. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130561 · Cited by 8
“Machine learning models are known to learn biases [67] and shortcuts [68] from the data that lead to better metrics without necessarily discovering the rules or representations desired by the user.”
Garcia-Atutxa, I., Villanueva‐Flores, F., Barrio, E. D., Sanchez-Villamil, J. I., Martínez-Más, J., & Bueno-Crespo, A. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1649746 · Cited by 12
Khairil, L., Benny, K. H. S., Jerry, J., Khatib, F. M., Ramli, M. D. C., & Kumar, S. (2026) · Pharmaceuticals · doi:10.3390/ph19060916 · Cited by 11
“In other words, AI is trained almost exclusively on successful results, making it difficult to understand what will fail because computationally promising candidates may fail during biological and clinical validation [50].”
COVID-19 Detection via Image Classification using Deep Learning on Chest X-Ray
Azeem, M. A., Khan, M. I., & Khan, S. A. (2021) · doi:10.1109/ee-rds53766.2021.9708588 · Cited by 8
WGS Data Collections: How Do Genomic Databases Transform Medicine?
Król, Z. J., Dobosz, P., Ślubowska, A., & Mroczek, M. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24033031 · Cited by 8
“The authors advise standardisation of methodology on several levels [115].”
BackgroundCosta, M. V. L., de Aguiar, E. J., Rodrigues, L. S., Ramos, J. S., Traina, C., & Traina, A. J. M. (2023) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms58004.2023.00272 · Cited by 8
Antony, M., Kakileti, S. T., Shah, R., Sahoo, S., Bhattacharyya, C., & Manjunath, G. (2023) · Scientific Reports · doi:10.1038/s41598-023-44653-y · Cited by 12
“A systematic review by Roberts et al. 45 found that machine learning models for COVID-19 detection using chest radiographs and CT scans were not of potential clinical use due to methodological flaws and biases.”
BackgroundPET Synthesis via Self-Supervised Adaptive Residual Estimation Generative Adversarial Network
Xue, Y., Bi, L., Peng, Y.-G., Fulham, M., Feng, D. D., & Kim, J. (2023) · IEEE Transactions on Radiation and Plasma Medical Sciences · doi:10.1109/trpms.2023.3339173 · Cited by 8
Fang, Y., Xing, X., Wang, S., Walsh, S. L. F., & Yang, G. (2024) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2024.102778 · Cited by 8
Self-citationCappelletti, L., Rekerle, L., Fontana, T., Hansen, P., Casiraghi, E., Ravanmehr, V., Mungall, C., Yang, J. J., Spranger, L., Karlebach, G., et al. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae036 · Cited by 8
Jony, S. S. R., Haque, U., Webb, N. J., Spence, E. E., Rahman, M. S., Aghamohammadi, N., Lie, Y., Ángulo-Molina, A., Ananth, S., Ren, X., et al. (2021) · Behavioral Sciences · doi:10.3390/bs11080106 · Cited by 8
O’Shea, A., Li, M., Mercaldo, N. D., Balthazar, P., Som, A., Yeung, T., Succi, M. D., Little, B. P., Kalpathy–Cramer, J., & Lee, S. I. (2022) · BJR|Open · doi:10.1259/bjro.20210062 · Cited by 8
COVID-19 in Italy: Comparison of CT Findings from Time Zero to the Delta Variant
Maggialetti, N., Villanova, I., Castrì, A., Greco, C. N., Inchingolo, F., Virgilio, D., Moschetta, M., Sardaro, A., Ianora, A. A. S., & Scardapane, A. (2022) · Microorganisms · doi:10.3390/microorganisms10040796 · Cited by 8
“In addition, today’s new techniques such as machine learning methods offer a great help in terms of the fast and accurate recognition of COVID-19 disease in both radiographs and CT images [11].”
MethodHan, X., Yu, Z., Zhuo, Y., Zhao, B., Ren, Y., Lamm, L., Xue, X., Feng, J., Marr, C., Shan, F., et al. (2021) · iScience · doi:10.1016/j.isci.2022.104227 · Cited by 8
“Based on CT images or clinical data or both artificial intelligence (AI)-based methods have shown advantages in disease diagnosis and progression prediction (Shi et al., 2020; Wong et al., 2020; Roberts et al., 2021; Zhu et al., 2021).”
Ragon, B., Volkov, B. B., Pulley, C., & Holmes, K. (2022) · Journal of Clinical and Translational Science · doi:10.1017/cts.2022.402 · Cited by 8
“…January to October 2020), describing new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images, Roberts et al. [34] found that “ none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases, ” which is “ a…”
BackgroundRidzuan, M., Bawazir, A., Navarrete, I. G., Almakky, I., & Yaqub, M. K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-12053-4_18 · Cited by 9
“Despite high reported accuracies by these methods, questions have been raised regarding their clinical usefulness due to the bias of small datasets, poor integration of multistream data, variability of international sources, difficulty of prognosis, and the lack of collaborative work between clinicians and data analys…”
BackgroundPersonalized survival probabilities for SARS-CoV-2 positive patients by explainable machine learning
Zucco, A. G., Agius, R., Svanberg, R., Moestrup, K. S., Marandi, R. Z., MacPherson, C. R., Lundgren, J. D., Ostrowski, S. R., & Niemann, C. U. (2022) · Scientific Reports · doi:10.1038/s41598-022-17953-y · Cited by 9
“Meta-analyses have indicated that the majority of published models suffer from a risk of bias due to overfitting, small sample sizes, poor cohort definition or not considering censored patients 18,29 .”
BackgroundNeural-Symbolic Ensemble Learning for early-stage prediction of critical state of Covid-19 patients
Fadja, A. N., Fraccaroli, M., Bizzarri, A., Mazzuchelli, G., & Lamma, E. (2022) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-022-02674-1 · Cited by 9
“Due to the current historical period and thanks to the wide availability of data, applying ML and DL to Covid-19 data is an active and ongoing area of research [7, 8].”
BackgroundUsing model explanations to guide deep learning models towards consistent explanations for EHR data
Watson, M., Hasan, B. A. S., & Moubayed, N. A. (2022) · Scientific Reports · doi:10.1038/s41598-022-24356-6 · Cited by 12
Detecting COVID-19 from Chest X-rays Using Convolutional Neural Network Ensembles
Lel, T. E., Ahsan, M., & Haider, J. (2023) · Computers · doi:10.3390/computers12050105 · Cited by 9
Automatic detection of COVID-19 and pneumonia from chest X-ray images using texture features
Sheikhi, F., Taghdiri, A., Moradisabzevar, D., Rezakhani, H., Daneshkia, H., & Goodarzi, M. (2023) · Journal of Supercomputing · doi:10.1007/s11227-023-05452-4 · Cited by 6
CoRSAI: A System for Robust Interpretation of CT Scans of COVID-19 Patients Using Deep Learning
Avetisian, M., Burenko, I., Egorov, K. S., Kokh, V., Nesterov, A. Y., Nikolaev, A., Ponomarchuk, A., Sokolova, E., Tuzhilin, A., & Umerenkov, D. (2021) · ACM Transactions on Management Information Systems · doi:10.1145/3467471 · Cited by 8
Good machine learning practices: Learnings from the modern pharmaceutical discovery enterprise
Makarov, V. A., Chabbert, C. D., Koletou, E., Psomopoulos, F. E., Kurbatova, N., Ramirez, S. A., Nelson, C., Natarajan, P., & Neupane, B. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108632 · Cited by 8
Applications of AI-Based Models in the Field of Biomedicine
Doke-Bagade, P. E., & Bagade, O. M. (2024) · Springer eBooks · doi:10.1007/978-981-97-2596-0_12 · Cited by 7
Chen, X., Huang, Y., Jessney, B., Sangha, J., Gu, S. Z., Schönlieb, C., Bennett, M. R., & Roberts, M. T. (2025) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztaf053 · Cited by 9
“28 comparing manuscripts with the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) 31 and excluding those papers, which failed any of eight mandatory criteria (detailed in the Supplementary Information).”
Self-citationChallenges and Strategies for Deep Learning in Cardiovascular Imaging
Pasdeloup, D., Østvik, A., Olaisen, S. H., Skogvoll, E., Dalen, H., & Løvstakken, L. (2025) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2025.02.011 · Cited by 8
Damir, S. S., Probst, J., deMello, A. J., & Stavrakis, S. (2025) · Chemistry - Methods · doi:10.1002/cmtd.202500069 · Cited by 10
Chang, L., Li, H., Wu, W., Liu, X., Yan, J., Chen, Z., Wu, H., & Song, S. (2025) · Journal of Translational Medicine · doi:10.1186/s12967-025-07591-z · Cited by 10
“…learn and amplify [179]; (iv) leakage risks (patient overlap across splits and refitting normalization on combined data) that overstate performance [180]; and (v) imbalanced outcomes (rare fusions and never-smoker subsets) that produce unstable thresholds and poor calibration in under-represented…”
The comfort of automation: why cognitive sovereignty matters in AI-driven life sciences
Branda, F., & Ciccozzi, M. (2026) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2026.100158 · Cited by 8
Scarpiniti, M., Ahrabi, S. S., Baccarelli, E., Piazzo, L., & Momenzadeh, A. (2021) · Applied Sciences · doi:10.3390/app11198867 · Cited by 7
“CT [74] CT [75] CXR + CT [65] CXR + CT [76] CXR + CT [77] CXR + CT [78] CXR + CT Segmentation [79] CXR [80] CT [81] CT [82] CT [56] CT [83] CXR [84] CXR CT [85] CT [86] CT [87] CXR [20] CXR [88] CXR Unsupervised [89] Histopathological [90] CT [91] CT [92] CXR CXR [95] Skin [96] CT [97] MRI [98]…”
BackgroundMethodA 178-clinical-center experiment of integrating AI solutions for lung pathology diagnosis
Ibragimov, B., Arzamasov, K. M., Maksudov, B., Kiselev, S., Mongolin, A., Mustafaev, T., Ibragimova, D., Evteeva, K., Андрейченко, А. Е., & Morozov, S. P. (2023) · Scientific Reports · doi:10.1038/s41598-023-27397-7 · Cited by 7
“Wynants et al. 3 and Roberts et al. 4 assessed the risk of bias and potential applicability of COVID-19 models using the PROBAST tool.”
BackgroundMultilevel Explainable Artificial Intelligence: Visual and Linguistic Bonded Explanations
Aysel, H. I., Cai, X., & Prügel‐Bennett, A. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3308555 · Cited by 8
“…ahighpredictiveperformanceindetectingCOVID-19cases,a TuringInstitute’srecentreport[7]disappointinglyfoundthat artificialintelligence(AI)usedtodetectcoronavirushadlittle tonobenefitandmayevenbeharmful,mainlyduetounnoticed biasesinthedataanditsinherentblack-boxnature(also,see, e.g.,[8]).”
Imagawa, K., & Shiomoto, K. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-00975-5 · Cited by 10
Interpretable vision transformer based on prototype parts for COVID‐19 detection
Xu, Y., & Meng, Z. (2024) · IET Image Processing · doi:10.1049/ipr2.13074 · Cited by 7
Artificial intelligence: help or hindrance in solving the reproducibility crisis?
Straiton, J. (2024) · BioTechniques · doi:10.1080/07366205.2024.2355776 · Cited by 8
Habli, I., Sujan, M. A., & Lawton, T. (2024) · Future Healthcare Journal · doi:10.1016/j.fhj.2024.100179 · Cited by 8
Generative AI and unstructured audio data for precision public health
Anibal, J., Landa, A., Nguyen, H. T. T., Daoud, V., Le, T., Huth, H. B., Song, M., Peltekian, A., Shin, A., Hazen, L., et al. (2025) · npj Health Systems · doi:10.1038/s44401-025-00022-7 · Cited by 7
Determination of critical decision points for COVID-19 measures in Japan
Kim, J., Matsunami, K., Okamura, K., Badr, S., & Sugiyama, H. (2021) · Scientific Reports · doi:10.1038/s41598-021-95617-z · Cited by 7
When AI is wrong: addressing liability challenges in women's healthcare
Marotta, A. (2022) · Journal of Computer Information Systems · doi:10.1080/08874417.2022.2089773 · Cited by 7
A Light Deep Learning Algorithm for CT Diagnosis of COVID-19 Pneumonia
Furtado, A. N., da Purificação, C. A. C., Badaró, R. J. D. S., & Nascimento, E. G. S. (2022) · Diagnostics · doi:10.3390/diagnostics12071527 · Cited by 7
“A study by [17] systematically reviewed publications of machine learning models for the diagnosis or prognosis of COVID-19 from X-ray or CT images, concluding that all identified models had methodological flaws and/or underlying biases preventing their use in clinical practice.”
MethodAzad, A. K., Mahabub-A-Alahi, M.-A.-A., Ahmed, I., & Ahmed, M. U. (2023) · Diagnostics · doi:10.3390/diagnostics13030574 · Cited by 8
COVID-19 Detection via Fusion of Modulation Spectrum and Linear Prediction Speech Features
Zhu, Y. M., Tiwari, A. S., Monteiro, J., Kshirsagar, S., & Falk, T. H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3265603 · Cited by 7
Palmisano, A., Gnasso, C., Cereda, A. F., Vignale, D., Leone, R., Nicoletti, V., Barbieri, S., Toselli, M., Giannini, F., Loffi, M., et al. (2023) · European Radiology · doi:10.1007/s00330-023-09702-0 · Cited by 8
“Despite their potential utility, these prediction scores were hardly applicable in clinical routine due to methodological flaws [6], including the number of variables needed (leading to potentially huge numbers of missing variables), the long turnaround time of required lab tests, and the lack of…”
BackgroundHuman-Curated Validation of Machine Learning Algorithms for Health Data
Boman, M. (2023) · Digital Society · doi:10.1007/s44206-023-00076-w · Cited by 7
“This is a general problem of such competitions (Masnick, 2012), but in healthcare, it might also contribute to frustration with inadequacy of AI solutions in the clinic, diminishing human trust in such algorithms (Roberts et al., 2021).”
BackgroundPreemptively pruning Clever-Hans strategies in deep neural networks
Linhardt, L., Müller, K., & Montavon, G. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102094 · Cited by 10
Zhu, Y. M., Imoussaïne-Aïkous, M., Côté‐Lussier, C., & Falk, T. H. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3390990 · Cited by 7
“As data distributions vary across datasets [72], diagnostics performance obtained under within-dataset conditions may lack external validity and has been shown to be over-optimistic [41], [44], [72].”
BackgroundHaque, S., Liu, A. W., Liu, S., & Chan, J. H. (2021) · Advances in Information Technology · doi:10.1145/3468784.3470655 · Cited by 7
“advised caution over the use of public repositories in COVID-19 detection from chest radiographs and CT scans [3].”
BackgroundThe Evolution of Artificial Intelligence in Medical Informatics: A Bibliometric Analysis
Penteado, B. E., Fornazin, M., & Castro, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86230-5_10 · Cited by 7
Kulikowski, C. A., & Maojo, V. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100438 · Cited by 7
A comparison of Covid-19 early detection between convolutional neural networks and radiologists
Albiol, A., Albiol, F., Paredes, R., Plasencia-Martínez, J. M., Barrio, A. B., Santos, J. M. G., Tortajada, S., Montaño, V. M. G., Godoy, C. R., Gómez, S. F., et al. (2022) · Insights into Imaging · doi:10.1186/s13244-022-01250-3 · Cited by 9
“The checklist for Artificial Intelligence in Medical Imaging (CLAIM) has been added as an Additional file 1 [17] as suggested by Roberts et al. in [18].”
MethodMachine learning in translation
(2021) · Nature Biomedical Engineering · doi:10.1038/s41551-021-00758-1 · Cited by 7
Awasthi, R., Mishra, S. V., Cywiński, J. B., Maheshwari, K. K., Khanna, A. K., Papay, F., & Mathur, P. (2023) · medRxiv · doi:10.1101/2022.12.31.22284092 · Cited by 7
Integrating digital health technologies into complex clinical systems
Sujan, M. A. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100885 · Cited by 7
Courtenay, L. A., Vanderesse, N., Doyon, L., & Souron, A. (2024) · Journal of Computer Applications in Archaeology · doi:10.5334/jcaa.145 · Cited by 7
Knowledge transfer, knowledge gaps, and knowledge silos in citation networks
Cunningham, E., & Greene, D. (2025) · PLoS ONE · doi:10.1371/journal.pone.0329302 · Cited by 7
“Limitations to the utility of these applications have been highlighted by recent studies [38, 46].”
InfluentialPourakbari, B., Mamishi, S., Valian, S. K., Mahmoudi, S., Sadeghi, R. H., Abdolsalehi, M. R., Khodabandeh, M., & Farahmand, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-15366-1 · Cited by 6
“Neural Networks were included due to their ability to model complex, non-linear relationships 42,43 .”
SERS Meets Artificial Intelligence: A New Frontier in Cancer Diagnosis and Prognosis
Huda, N. U., Bari, R. Z. A., Javed, M., Kiani, M. N., & Jin, Y. (2026) · Analytical Chemistry · doi:10.1021/acs.analchem.6c00874 · Cited by 6
Zhang, L., Shu, J., Hu, J., Li, F., He, J., Wang, P., & Shen, Y. (2024) · Electronics · doi:10.3390/electronics13112002 · Cited by 6
Agrawal, R., Tofighbakhsh, A., Davidson, J., & Gabriele, J. (2025) · Current Surgery Reports · doi:10.1007/s40137-024-00442-8 · Cited by 6
“Education on how to identify and mitigate bias is also important [78–80].”
AI-Empowered Computational Examination of Chest Imaging for COVID-19 Treatment: A Review
Deng, H., & Li, X. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.612914 · Cited by 6
“(Dong et al. (2021) and Roberts et al. (2021) have broadly covered the use of various medical imaging modalities for COVID-19 treatment and Shi et al. (2021) overviewed all aspects along the chest imaging pipeline, from imaging data acquisition to image segmentation and diagnosis.”
BackgroundThe Future of Artificial Intelligence for Alzheimer’s Disease Diagnostics
Logan, R. A., Zerbey, S. S., & Miller, S. (2021) · Advances in Alzheimer s Disease · doi:10.4236/aad.2021.104005 · Cited by 6
“For example, when AI was applied in the clinic for COVID-19 diagnostics based on chest radiographs and CT scans, it failed miserably [23].”
BackgroundMethodZhou, Y., Liu, J., Yang, Z., Liu, T., Meng, X., Zhou, Z., Anaissi, A., & Braytee, A. (2022) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw58026.2022.00054 · Cited by 6
“Second, many previous studies used poor-quality data and had high bias due to imbalanced datasets that were obtained from public sources [5].”
BackgroundResultWhy is biomedical informatics hard? A fundamental framework
Johnson, T. R., & Bernstam, E. V. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104327 · Cited by 6
Smith, G. (2023) · doi:10.1093/oso/9780192868459.001.0001 · Cited by 6
Artificial Intelligence, Bioinformatics, and Pathology
Levy, J., Lu, Y., Montivero, M. K. S., Ramwala, O. A., McFadden, J. R., Miles, C., Diamond, A. G., Reddy, R., Reddy, R., Hudson, T. R., et al. (2022) · Advances in Molecular Pathology · doi:10.1016/j.yamp.2023.01.001 · Cited by 7
Socha, M., Prażuch, W., Suwalska, A., Foszner, P., Tobiasz, J., Jaroszewicz, J., Gruszczyńska, K., Sliwinska, M., Nowak, M., Giżycka, B., et al. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107684 · Cited by 6
Dadras, A. A., Jaziri, A., Frodl, E., Vogl, T. J., Dietz, J., & Bucher, A. M. (2023) · Bioengineering · doi:10.3390/bioengineering11010021 · Cited by 6
“In order to prevent quality issues in experiments [22] we followed an established reporting guideline to keep track of the imaging setting [1].”
MethodShiri, I., Salimi, Y., Saberi, A., Pakbin, M., Hajianfar, G., Avval, A. H., Sanaat, A., Akhavanallaf, A., Mostafaei, S., Mansouri, Z., et al. (2024) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.23028 · Cited by 31
“35 In addition, most studies have not provided sufficient information regarding data preprocessing and demographics of training/testing cohorts and did not provide public access to code/data.”
BackgroundMethodInfluentialStrotzer, Q., Wagner, T., Angstwurm, P., Hense, K., Scheuermeyer, L., Noeva, E., Dinkel, J. G., Stroszczynski, C., Fellner, C., Riemenschneider, M. J., et al. (2024) · Neuro-Oncology Advances · doi:10.1093/noajnl/vdae060 · Cited by 6
“30 Another review examining the methodological quality of developed AI models from a wide range of disciplines concluded that a significant proportion of developers do not separate train/validation from test sets during preprocessing, leading to data leakage.”
BackgroundTeliNet: Classifying CT scan images for COVID-19 diagnosis
Teli, M. N. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00060 · Cited by 7
“Many of these approaches specifically use images obtained from Chest Radiographs (CXR) and chest Computed Tomography(CT) [26].”
BackgroundMethodSaba-Sadiya, S., Alhanai, T., & Ghassemi, M. M. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746397 · Cited by 6
Clinical Artificial Intelligence
McDermott, M. B. A., Nestor, B., & Szolovits, P. (2022) · Clinics in Laboratory Medicine · doi:10.1016/j.cll.2022.09.004 · Cited by 7
D-Cov19Net: A DNN based COVID-19 detection system using lung sound
Chatterjee, S., Roychowdhury, J., & Dey, A. (2022) · Journal of Computational Science · doi:10.1016/j.jocs.2022.101926 · Cited by 6
Factors determining generalization in deep learning models for scoring COVID-CT images
Horry, M. J., Chakraborty, S., Pradhan, B., Fallahpoor, M., Chegeni, H., & Paul, M. (2021) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2021456 · Cited by 13
The BJUI Editorial Team's view on artificial intelligence and machine learning
Thalmann, G. N., Klatte, T., Papa, N. P., & Carlsson, S. (2023) · British Journal of Urology · doi:10.1111/bju.16024 · Cited by 6
“In a recent review of 62 studies that used AI to diagnose COVID-19 from medical scans, Roberts et al. [5] found that none of the models were ready to be deployed clinically for use in diagnosing or predicting the prognosis of COVID-19, because of fl aws such as biases in the data, methodology…”
MethodNavigating the unseen peril: safeguarding medical imaging in the age of AI
Maertens, A., Brykman, S., Härtung, T., Gafita, A., Bai, H. X., Hoelzer, D., Skoudis, E., & Paller, C. J. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1400732 · Cited by 6
Metadata
- Authors
- Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., Weir-McCall, J. R., Teng, Z., Gkrania-Klotsas, E., AIX-COVNET, Ruggiero, A., Korhonen, A., Jefferson, E., Ako, E., Langs, G., Gozaliasl, G., Yang, G., Prosch, H., Preller, J., Stanczuk, J., Tang, J., Hofmanninger, J., Babar, J., Sánchez, L. E., Thillai, M., Gonzalez, P. M., Teare, P., Zhu, X., Patel, M., Cafolla, C., Azadbakht, H., Jacob, J., Lowe, J., Zhang, K., Bradley, K., Wassin, M., Holzer, M., Ji, K., Ortet, M. D., Ai, T., Walton, N., Lio, P., Stranks, S., Shadbahr, T., Lin, W., Zha, Y., Niu, Z., Rudd, J. H. F., Sala, E., & Schönlieb, C.-B.
- Affiliations
- AstraZeneca (United Kingdom) (GB)
- University of Cambridge (GB)
- University of Manchester (GB)
- Royal Papworth Hospital (GB)
- University of Dundee (GB)
- Royal Brompton Hospital (GB)
- Medical University of Vienna (AT)
- University of Helsinki (FI)
- Imperial College London (GB)
- Boehringer Ingelheim (Germany) (DE)
- Addenbrooke's Hospital (GB)
- University College London (GB)
- Macau University of Science and Technology (MO)
- Tongji Hospital (CN)
- Renmin Hospital of Wuhan University (CN)
- Corresponding author
- Roberts, Michael
- ORCID
- Roberts, Michael
- Thorpe, Matthew
- Gilbey, Julian
- Yeung, Michael
- Ursprung, Stephan
- McCague, Cathal
- Weir-McCall, Jonathan R.
- Gkrania-Klotsas, Effrossyni
- Ako, Emmanuel
- Langs, Georg
- Gozaliasl, Ghassem
- Yang, Guang
- Prosch, Helmut
- Preller, Jacobus
- Azadbakht, Hojjat
- Jacob, Joseph
- Ji, Kangyu
- Walton, Nicholas
- Stranks, Samuel
- Shadbahr, Tolou
- Rudd, James H. F.
- Sala, Evis
- Schönlieb, Carola-Bibiane
- Venue
- Nature Machine Intelligence
- Published
- 2021-03-15
- Volume
- 3
- Issue
- 3
- Pages
- 199-217
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W3119527628
- Cited by
- 1,044
- References
- 83
- Citations by year
- 2026: 203 · 2025: 96 · 2024: 143 · 2023: 248 · 2022: 224 · 2021: 130
- FWCI
- 98.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Machine Intelligence (publishedVersion, cc-by)
- Discovery Research Portal (University of Dundee) (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- elib (German Aerospace Center) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Työväentutkimus Vuosikirja (submittedVersion)
- arXiv (Cornell University)
- Funders
- AstraZeneca
- European Commission (777826)
- Medical Research Council (1966157, MR/S010351/1)
- Engineering and Physical Sciences Research Council (EP/T017961/1, EP/N014588/1, EP/S026045/1)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- COVID-19 pandemic, machine learning, chest radiograph, chest CT, COVID-19 detection, outcome prediction, clinical utility, model validation, methodological errors, bias, research recommendations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
