The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.
Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
Open accessAbstract
Abstract Machine learning methods offer great promise for fast and accurate detection and prognostication of coronavirus disease 2019 (COVID-19) from standard-of-care chest radiographs (CXR) and chest computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we consider all published papers and preprints, for the period from 1 January 2020 to 3 October 2020, which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. All manuscripts uploaded to bioRxiv, medRxiv and arXiv along with all entries in EMBASE and MEDLINE in this timeframe are considered. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher-quality model development and well-documented manuscripts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Much of the current evidence is based on retrospective analyses with limited prospective validation [19,28].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The success of deep learning in computer vision (CV) analysis has significantly advanced its clinical applications in comprehending biomedical images, including CT scans [88], positron emission tomography (PET)-CT scans [89], whole-slide images (WSIs) [90], and MRI scans [91].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [88]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Based on this study, we suggest that such a gold rush may give rise to un-engaged 'augmentation' which is highly risky from a learning and knowledge perspective for experts, AI companies, and consumers.”
From To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis (2022) · cites this paper as Roberts et al. 2021
In the citing paper
“Despite problems like the one discovered by Roberts et al. [195], the fact that ML systems process and analyze data differently from human beings can be a great benefit to science and society because these systems may be able to identify useful and innovative structures, properties, patterns, and relationships that human beings would not recognize.”
From The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool (2025) · cites this paper as [195]
In the citing paper
“Differences in image acquisition and processing may result in “shortcuts” where models take cues from non-clinical features.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“For example, COVID-19 prediction models have been demonstrated to learn which institution the images were obtained from rather than features pertaining to the underlying disease. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“The methodological flaws likely is a reflection of the lack of diversity in the development team to allow clinicians, data analysts and patient advocacy groups to work side-by-side throughout the AI life cycle. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“A similar result is summarized in a systematic review of COVID-19 prediction models on CT scans and Chest X-rays (CXRs) that found limited clinical utility due to methodological flaws and underlying biases stemming from small training data sets, data set variability, and limited integration of non-imaging data, among others.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Wang, D., Hu, B., Hu, C., Zhu, F., Liu, X., Zhang, J., Wang, B., Xiang, H., Cheng, Z., Xiong, Y., et al. (2020) · JAMA · doi:10.1001/jama.2020.1585 · Cited by 20,148
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Ai, T., Yang, Z., Hou, H., Zhan, C., Chen, C., Lv, W., Tao, Q., Sun, Z., & Xia, L. (2020) · Radiology · doi:10.1148/radiol.2020200642 · Cited by 5,793
Self-citationRadiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688
Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning
Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C. S., Liang, H., Baxter, S. L., McKeown, A., Yang, G., Wu, X., Yan, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.010 · Cited by 4,842
Self-citation
Research works printed but not linked: 7
- Wong, H. Y. F. et al. Frequency and distribution of chest radiographic findings in COVID-19 positive patients. Radiology 296, 201160 (2019).
- Bassi, P. R. A. S. & Attux, R. A deep convolutional neural network for COVID-19 detection using chest X-rays. Preprint at http://arxiv.org/abs/2005.01578 (2020). · arXiv
- Amyar, A., Modzelewski, R. & Ruan, S. Multi-task deep learning based CT imaging analysis for COVID-19: classification and segmentation. Preprint at medRxiv https://doi.org/10.1101/2020.04.16.2006470 (2020). · DOI
- Goodfellow, I. J. et al. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems Vol. 2 2672–2680 (MIT Press, 2014).
- Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations ICLR 2017 (ICLR, 2017).
- Covidence Systematic Review Software v2486 6284093b (Veritas Health Innovation, 2020); https://www.covidence.org/
- Aviles-Rivero, A. I., Sellars, P., Schönlieb, C.-B. & Papadakis, N. GraphXCOVID: explainable deep graph diffusion pseudo-labelling for identifying COVID-19 on chest X-rays. Preprint at https://arxiv.org/abs/2010.00378 (2020). · arXiv
Cited by
Sorted by citations
Items 701–800 of 1,000
Aquino-García, A. M., Pérez-Espinosa, H., Andreu-Pérez, J., & Rodríguez‐González, A. Y. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8030080 · Cited by 1
“Systematic evaluations have highlighted common pitfalls related to dataset bias, lack of external validation, and potential overestimation of performance [25].”
Olorede, K. O., & Ade-Ibijola, A. (2026) · ADCAIJ ADVANCES IN DISTRIBUTED COMPUTING AND ARTIFICIAL INTELLIGENCE JOURNAL · doi:10.14201/adcaij.32890 · Cited by 1
“The COVID-19 pandemic (Keesara et al., 2020; Roberts et al., 2021), spanning 2020 to 2022, further accelerated AI adoption in essential services, especially in public health contexts.”
Early Fusion of Texture-Encoded CT with CXR Images for Pulmonary Disease Classification
Shingan, G. G., Jagtap, A. M., & Chaudhary, S. (2026) · SN Computer Science · doi:10.1007/s42979-026-04899-5 · Cited by 1
Masoomi, M. A., Alkandari, L., Ramzy, H., & Hamza, M. A. (2026) · Diagnostics · doi:10.3390/diagnostics16070995 · Cited by 1
“Macro-averaged metrics were computed as the unweighted mean of class-wise values, while weighted metrics were calculated as the mean of class-wise values weighted by class support, consistent with methodological guidance for multi-class classification [24].”
InfluentialKiso, T., Okada, Y., Kawata, S., Shichiji, K., Okumura, E., Hatsumi, N., Matsuura, R., Kaminaga, M., Kuwano, H., Okumura, E., et al. (2026) · BMC Medical Imaging · doi:10.1186/s12880-026-02249-8 · Cited by 1
“…limited (e.g., on the scale of a few hundred examples), the “relative weight” of the non-image data is smaller than the rich feature set that the CNN extracts from the images, and the possibility that the data may be buried in image-derived signals, even after integration cannot be ruled out [44].”
Zehra, I., Ali, S., & Lee, S. W. (2026) · Journal for Healthcare Quality · doi:10.1097/jhq.0000000000000524 · Cited by 3
Harms, R. J., Ankeny, R. A., Carter, L., Mankad, A., & Scully, J. L. (2026) · AI and Ethics · doi:10.1007/s43681-026-01088-1 · Cited by 1
Alwerfali, M. A. E., Faraji, F., Abdalqadir, M. O., & Ali, J. A. (2026) · International Journal of Energy Research · doi:10.1155/er/2452722 · Cited by 1
Yankam, B. M. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1739192 · Cited by 1
“Meta-analyses of COVID-19 diagnostic models showed that complete diagnostic tables were available in only nine of 62 studies (Roberts et al., 2021), showing the fragility of current evidence synthesis.”
Cobo, V. T., Robles-Garcia, J. E., Guillén‐Grima, F., Lopez, A. C., Tapia-Tapia, M., Huerta, L. L., Ancizu-Marckert, F. J., Guillén-Aguinaga, L., Sánchez-Zalabardo, D., & López, B. M. (2026) · Diagnostics · doi:10.3390/diagnostics16101454 · Cited by 1
“Stage 4 should carry out multi-centre external validation, using prospective or pre-registered retrospective cohorts, with pre-specified reporting of subgroup performance and calibration metrics in addition to discrimination [77,78].”
Artificial Intelligence in Anorexia Nervosa Care: Comparing ChatGPT and Google Gemini
Witkowska, W., & Bzikowska‐Jura, A. (2026) · Nutrients · doi:10.3390/nu18111705 · Cited by 1
“ML has been widely applied across multiple areas of medicine, including radiology and imaging-based diagnostics, clinical data analysis, predictive modeling, and personalized treatment planning [23].”
Qasem, M. H., Sariera, T. M. A., Alhumaid, K. F., Alshraah, S. M., Mufleh, A. S. S., & Chihaoui, N. (2026) · Journal of Imaging · doi:10.3390/jimaging12060228 · Cited by 1
“Closing this gap requires multicenter cohorts, prospective evaluation, and clinically relevant endpoints beyond accuracy [3,27,63].”
Trustworthy Assessment of University Competitiveness Using a Neural Network Model
Grzeszczyk, T. A. (2026) · Information · doi:10.3390/info17060536 · Cited by 1
Lai, H., Sousa‐Pinto, B., Cao, C., Moher, D., Estill, J., Liu, J., Zhao, W., Wang, Y., Ye, Z., Tong, B., et al. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112383 · Cited by 3
Navigating AI and machine learning in cancer research: an end-to-end translational framework
Saha, S., Ali, M. S., Tengli, A. K., Prasad, S. R., Mallikarjunaswamy, P., Pillappan, R., & Javarappa, K. K. (2026) · Journal of Translational Medicine · doi:10.1186/s12967-026-08503-5 · Cited by 3
“A systematic review of COVID-19 imaging AI found none of 415 evaluated models clinically deployable, with data leakage, site-confounded labels, and demographic bias as recurrent failure mechanisms [33].”
InfluentialTunable band-stop photodetection with machine learning-enabled broadband spectral adaptation
Yu, H., Lian, Y., Jia, S., Ma, W., Fu, J., Liu, X., Zhu, X., Zhang, L., Han, T., Lan, C., et al. (2026) · Nature Sensors · doi:10.1038/s44460-026-00098-0 · Cited by 1
Self-citationMedical question answering: A comprehensive multimodal and LLM-driven survey
Mhedhbi, E., Zhu, X., Ayaz, M., Khan, M., & Othmani, A. (2026) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2026.109541 · Cited by 1
Self-citationSuantai, S., & Janngam, K. (2026) · AIMS Mathematics · doi:10.3934/math.2026834 · Cited by 1
Lin, Z., Guo, Q., Jin, H., Huang, J., Wang, P., & Ouyang, R. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3397-7_3 · Cited by 1
Vallée, A. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001610 · Cited by 1
“Sparse laboratory trajectories, irregular sampling in electronic health records, and snapshot imaging data may be insufficient to reliably constrain dynamical systems models [52,53].”
Chhetri, L., & Kumar, A. (2026) · IEEE Guwahati Subsection Conference (GCON) · doi:10.1109/gcon69192.2026.11649062 · Cited by 1
“Roberts et al. [10] quantified the accuracy increase of image-level splitting over patient-level splitting to be 5-30 percent for several medical imaging tasks.”
Chhetri, L., Datta, A., & Ghosal, P. (2026) · IEEE Guwahati Subsection Conference (GCON) · doi:10.1109/gcon69192.2026.11648823 · Cited by 1
“Recent studies have shown that this flaw in methodology artificially boosts accuracy by 5-30 percentage points [7], [8], thereby rendering the results invalid.”
Toma, M., & YUSUPOV, D. (2026) · Bioengineering · doi:10.3390/bioengineering13091034 · Cited by 1
Li, J., Xiao, L., Qu, X., Du, L., Gong, T., Yu, Y., Li, Y., 谢鹏飞, Chu, G., Li, H., et al. (2026) · Radiology · doi:10.1148/radiol.253579 · Cited by 1
Artificial intelligence in biomarker discovery for diseases: diagnostic and therapeutic prospects
Dinç, R., & Ardıç, N. (2026) · Signal Transduction and Targeted Therapy · doi:10.1038/s41392-026-02946-4 · Cited by 1
“139,140 In contrast, approaches that incorporate network constraints, pathway priorities, or learning based on perturbation information are better positioned to identify features that remain stable across biological contexts.”
Tang, W., Wang, F., Wang, J., Huang, Y., Liu, L., Zhao, S., Zhao, X.-M., & Wu, N. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.654754 · Cited by 1
“Machine-learning based methods are emerging rapidly in recent years focusing on COVID detection or disease diagnosis (24, 25).”
BackgroundMethodHaque, S., & Chan, J. H. (2021) · doi:10.1145/3486713.3486735 · Cited by 1
“Many of these problems stem from the challenges of getting access to high-quality datasets that have been expertly annotated by multiple radiologists due to time and resources [20].”
BackgroundQuality Monitoring of Federated Covid-19 Lesion Segmentation
González, C., Harder, C. L., Ranem, A., Fischbach, R., Kaltenborn, I., Dadras, A. A., Bucher, A. M., & Mukhopadhyay, A. (2022) · Informatik aktuell · doi:10.1007/978-3-658-36932-3_8 · Cited by 1
Micallef, N., Debono, C. J., Seychell, D., & Attard, C. (2022) · IEEE Mediterranean Electrotechnical Conference (MELECON) · doi:10.1109/melecon53508.2022.9843100 · Cited by 1
RETRACTED: An unsupervised approach for personalized RHM with reduced mean alert latency
Marimuthu, P., & Vaidehi, V. (2022) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-220539 · Cited by 1
“Roberts et al. [17] done a study on covid 19 detection using CXR and CT images with machine learning.”
RetractedAI and the Infectious Medicine of COVID-19
Andriasyan, V., Petkidis, A., & Yakimovich, A. (2022) · Springer eBooks · doi:10.1007/978-3-031-08506-2_6 · Cited by 1
A Case Study of Using Machine Learning Techniques for COVID-19 Diagnosis
Dinacci, M., Chen, T., Mahmud, M., & Parkinson, S. (2022) · Brain informatics and health · doi:10.1007/978-981-19-5272-2_10 · Cited by 1
Omnidirectional 2.5D representation for COVID-19 diagnosis using chest CTs
da Silveira, T. L. T., Pinto, P. G. L., Lermen, T. S., & Jung, C. R. (2023) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2023.103775 · Cited by 1
“RT-PCR may not be widely available, is prone to false-negative scoring [6–9], and imaging exams like radiography and computed tomography (CT) allow detecting advanced SARS-CoV-2 pulmonary infections [3,10,11].”
InfluentialHo, A. (2023) · Oxford University Press eBooks · doi:10.1093/med/9780197556269.003.0001 · Cited by 1
Evaluating Generalizability of Deep Learning Models Using Indian-COVID-19 CT Dataset
Suba, S., Parekh, N., Loganathan, R., Pudi, V., & Sunkavalli, C. (2023) · Advances in health sciences research/Advances in Health Sciences Research · doi:10.2991/978-94-6463-164-7_3 · Cited by 1
Implementation of Artificial Intelligence in Primary Care
Yang, H. (2023) · Korean Journal of Family Practice · doi:10.21215/kjfp.2023.13.2.73 · Cited by 1
COVID-19 detection and classification: key AI challenges and recommendations for the way forward
Albatoul, A., Mirza, A. A., Sherin, A., Nouh, T. A., Mahboub, B., Salameh, L. I. J., Alkubeyyer, M., & Alsalamah, S. (2023) · Journal of Pulmonology and Respiratory Research · doi:10.29328/journal.jprr.1001044 · Cited by 1
Automated Reporting of Medical Diagnostic Imaging for Early Disease and Aging Biomarkers Detection
Андрейченко, А. Е., & Morozov, S. P. (2023) · Healthy ageing and longevity · doi:10.1007/978-3-031-35176-1_2 · Cited by 1
Proceedings of the Association for Pathology Informatics Bootcamp 2022
Obstfeld, A. E., Brodsky, V., Carter, A. B., Gershkovich, P., Haymond, S., Levy, B. P., Sinard, J. H., Sellers, D., Stoffel, M., & Jackups, R. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100331 · Cited by 1
Optimal control strategy for COVID-19 developed using an AI-based learning method
Kakulapati, V., & Jayanthiladevi, A. (2023) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2023.2256733 · Cited by 1
“(Driggs et al., 2021) explored the potential of machine learning techniques for COVID-19 diagnosis and prognosis.”
Medical Decision-Making and Artificial Intelligence
Djulbegović, B., & Hozo, I. (2023) · Cancer treatment and research · doi:10.1007/978-3-031-37993-2_9 · Cited by 1
Neurological Properties to Circumvent AI’s Error Reduction Impasse
Kobylarz, T. J., & Kobylarz, E. J. (2023) · Trends in Computer Science and Information Technology · doi:10.17352/tcsit.000070 · Cited by 2
Optimization of Operating Rooms Using Machine Learning Models and Artificial Intelligence
Lahoti, S. (2023) · International Conference on Smart Electronics and Communication (ICOSEC) · doi:10.1109/icosec58147.2023.10276178 · Cited by 1
“Out of 2,112 models, none could be used in possible clinical situations, according to [23], and it was found that the majority of them had empirical issues or fundamental biases.”
BackgroundMI-CLAIM Checklist Modelling for Clinical Artificial Intelligence
Dhawan, A. (2023) · International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC) · doi:10.1109/i-smac58438.2023.10290557 · Cited by 1
“A growing range of research in medical imaging shows that AI tools can significantly improve the reliability of prognostic predictions or the accuracy of diagnostics [8].”
BackgroundAttention-CNN Model for COVID-19 Diagnosis Using Chest CT Images
Suba, S., & Parekh, N. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-45170-6_43 · Cited by 1
Conformité européenne des systèmes d’IA : outils statistiques élémentaires
Besse, P. C. (2022) · Statistique et société · doi:10.4000/statsoc.565 · Cited by 1
AI-assisted Analysis to Facilitate Detection of Humeral Lesions on Chest Radiographs
Kim, H., Kim, K., Oh, S. J., Lee, S., Woo, J. H., Kim, J. H., Ki, Y., Kim, K., & Chung, M. J. (2024) · Radiology Artificial Intelligence · doi:10.1148/ryai.230094 · Cited by 2
Machine learning in mental health and its relationship with epidemiological practice
DelPozo‐Baños, M., Stewart, R. J., & John, A. M. (2024) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2024.1347100 · Cited by 5
Artificial Intelligence Applications in Pancreatic Cancer Imaging
Ahmed, T. M., Lopez‐Ramirez, F., Fishman, E. K., & Chu, L. C. H. (2024) · Advances in Clinical Radiology · doi:10.1016/j.yacr.2024.04.003 · Cited by 2
Adebiyi, A., Adhikari, P., Rao, P. R., & Ching, W. Y. (2024) · BME Horizon · doi:10.37155/2972-449x-vol2(1)-110 · Cited by 1
Machine Learning Applications in Prediction Models for COVID-19: A Bibliometric Analysis
Lv, H., Liu, Y., Yin, H., Xi, J., & Wei, P. (2024) · Information · doi:10.3390/info15090575 · Cited by 1
“Roberts et al. [59] argued that none of the reviewed studies were sufficient to transition from scientific research to clinical practice due to issues such as dataset bias, inadequate model evaluation, limited generalizability, and lack of reproducibility.”
Can AI shake-up translational research?
Wells, S. (2024) · Nature · doi:10.1038/d41586-024-03318-0 · Cited by 1
Attribute Relevance Score: A Novel Measure for Identifying Attribute Importance
Neirz, P., Allende, H., & Saavedra, C. (2024) · Algorithms · doi:10.3390/a17110518 · Cited by 1
“These issues become particularly evident in studies where models achieve high predictive accuracy by leveraging spurious correlations in the training data [25,26].”
Preprinting Does Not Meet Science’s Duty of Care Responsibility to Society
Browman, H. I., & Leopold, S. S. (2025) · Journal of Trial and Error · doi:10.36850/b6fa-4513 · Cited by 1
Ethical, Regulatory, and Practical Challenges in Artificial Intelligence-Driven Chest Imaging
de Lima, J. B. C., Griffin, I., Gemmell, J. S., Davis, K., Amanamba, U., Zanjani, N. A., Siyanaki, M. R. H., Mohammed, T.-L. H., Benos, T., Rodrigues, R. S., et al. (2025) · Seminars in Roentgenology · doi:10.1053/j.ro.2025.06.003 · Cited by 2
Khan, A. H., Li, J., Asghar, M. N., & Iqbal, S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0327419 · Cited by 1
“These advancements include the detection of chest diseases [20], the identification of cancer cells [21], the classification of brain and breast tumors [22], and the analysis of genomic sequences [23].”
RetractedFrom lab to pocket: A novel continual learning-based mobile application for screening COVID-19
Falero, D., Kabir, M. A., & Homaira, N. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112960 · Cited by 3
Moya-Sánchez, E. U., Sánchez-Pérez, A., Zarate-Macías, A., Nanclares, R., Villarreal-Padilla, E., Castillero, P. M., Sánchez-Montes, A., Jáuregui-Ulloa, E. E., Moreno, H. J., & Cortés, U. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00405-y · Cited by 1
“One approach to address these challenges is to apply a responsible AI framework.”
Altaf, F., Islam, S. M. S., & Akhtar, N. (2021) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta52665.2021.9647158 · Cited by 1
Fischer, G., Silvestro, A. D., Müller, M., Frauenfelder, T., & Martini, K. (2021) · Academic Radiology · doi:10.1016/j.acra.2021.09.016 · Cited by 1
On Robust Economic Control of Epidemics With Application to COVID-19
Hsieh, C.-H. (2021) · IEEE Access · doi:10.1109/access.2021.3136191 · Cited by 1
COVID-Net Network and Application on Support Diagnosis COVID-19 over X-ray Images
Do, T. H., Le, H., & Ha, T.-H. (2022) · International Conference on Multimedia Analysis and Pattern Recognition (MAPR) · doi:10.1109/mapr56351.2022.9924841 · Cited by 1
“Several investigations [20], [23], [24] have looked into imaging findings identified in CXR pictures of SARS-CoV-2 positive individuals, with bilateral abnormalities, ground-glass opacity, and interstitial abnormalities being the most prevalent [20].”
BackgroundIntroduction: Trends, Puzzles, and Hopes for the Future of Healthcare
Ehsani, S., Glauner, P., Plugmann, P., & Thieringer, F. M. (2022) · Future of business and finance · doi:10.1007/978-3-030-99838-7_1 · Cited by 1
Abubakar, H., & Idris, M. (2023) · Artificial intelligence · doi:10.5772/intechopen.106210 · Cited by 1
Performance Analysis of Machine Learning Algorithms for COVID-19 Detection
Thakare, A., Gulhane, P. G., Chaudhari, S., & Baradkar, H. (2022) · IEEE Bombay Section Signature Conference (IBSSC) · doi:10.1109/ibssc56953.2022.10037516 · Cited by 1
“A research and performance comparison of ten well-known convolution neural networks is conducted [12].”
Smith, G. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780192868459.003.0002 · Cited by 1
Assessing robustness of a deep-learning model for COVID-19 classification on chest radiographs
Shenouda, M., Kaveti, A., Flerlage, I., Kalpathy–Cramer, J., Giger, M. L., & Armato, S. G. (2023) · SPIE Medical Imaging: Computer-Aided Diagnosis · doi:10.1117/12.2652106 · Cited by 1
“To address this lack of generalizability, future work will investigate (1) the impact of sex on the classifier, (2) whether there were differences in the negative or positive patients between the two cohorts, and (3) altering the architecture of the model to make it more robust.”
BackgroundMethodFew-Shot Image Classification for Automatic COVID-19 Diagnosis
Cores, D., Vila-Blanco, N., Mucientes, M., & Carreira, M. J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36616-1_43 · Cited by 2
5. Information and Engineering
Ingram, D. (2023) · Open Book Publishers · doi:10.11647/obp.0335.05 · Cited by 1
OpenClinicalAI: enabling AI to diagnose diseases in real-world clinical settingsPreprint
Huang, Y., Wang, N., Tang, S., Ma, L., Hao, T., Jiang, Z., Zhang, F., Kang, G., Miao, X., Guan, X., et al. (2021) · medRxiv · doi:10.1101/2021.09.08.21263249 · Cited by 2
Fang, L., Green, J. W., Ye, X., & Shi, W. (2022) · Journal of Planning Education and Research · doi:10.1177/0739456x221128501 · Cited by 1
“This can lead to mis-taken interpretation of results, or the misuse and abuse of these techniques (Rey 2021; Roberts et al. 2021).”
Cappelletti, L., Rekerle, L., Fontana, T., Hansen, P., Casiraghi, E., Ravanmehr, V., Mungall, C., Yang, J. J., Spranger, L., Karlebach, G., et al. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.11.21.517376 · Cited by 1
“One reason for this might be that published models not infrequently display methodological flaws or underlying biases [6, 7, 8, 9].”
BackgroundEmam, K. E., & Malin, B. (2022) · JMIR AI · doi:10.2196/42046 · Cited by 1
“With some notable exceptions [1], and despite many publications, the impact of AI/ML in practice has been limited [2,3].”
BackgroundSrivastava, M., Reinhart, A., & Mejia, R. (2023) · medRxiv · doi:10.1101/2023.02.03.23285405 · Cited by 1
Casey, A., Ansari, S., Nakisa, B., Kelly, B., Brown, P., Cooper, P., Muhammad, I., Livingstone, S., Reddy, S., & Mäkinen, V. (2023) · medRxiv · doi:10.1101/2023.02.23.23286374 · Cited by 1
“…is suggested that while the AI models have potential, they are generally unsuitable for clinical use and, if deployed prematurely, could lead to undesirable outcomes including stress on both patients, unnecessary intrusive procedures, and the health system and even death due to misdiagnosis [5,7].”
BackgroundDeep Active Learning for Robust Biomedical SegmentationPreprint
Arikan, M., Sallo, F. B., Montesel, A., Ahmed, H., Hagag, A. M., Book, M., Faatz, H., Cicinelli, M. V., Meshkinfamfard, S., Ongun, S., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.03.28.534521 · Cited by 1
“Deep learning (DL) for medical applications has shown great promise in experimental settings, but results have not generalized broadly, resulting in poor performance on real-world validation tests[2, 12, 4].”
BackgroundBabu, L. S. P., Kumar, S. S., Mohan, N., Krishankumar, R., Ravichandran, K. S., Senapati, T., & Sikha, O. K. (2023) · Research Square · doi:10.21203/rs.3.rs-3357602/v1 · Cited by 1
Ingram, D. (2023) · Open Book Publishers · doi:10.11647/obp.0335.04 · Cited by 1
AUGMENT: a framework for robust assessment of the clinical utility of segmentation algorithmsPreprint
McCague, C., Buddenkotte, T., Sanchez, L. E., Hulse, D., Pintican, R. M., Rundo, L., AUGMENT study team, Brenton, J. D., Couturier, D.-L., Öktem, O., et al. (2024) · medRxiv · doi:10.1101/2024.09.20.24313970 · Cited by 1
“Research groups developing medical image analysis tools often lack clinical experts, which can lead to basic errors that are obvious to clinicians being made and substantial development time being invested in tools that are ultimately not viable for clinical application(10).”
BackgroundSelf-citationShuaibu, I., Khan, M., Alkhamis, D., & Alkhamis, A. (2026) · medRxiv · doi:10.64898/2026.01.17.26344327 · Cited by 2
Patil, S. S., Jha, G. K., Mohsin, N., Patel, P., Ralte, L., & Shukla, A. K. (2026) · Cureus · doi:10.7759/cureus.109905
“Much of the current evidence is based on retrospective analyses with limited prospective validation [19,28].”
Quantifying Variability in Microscopy Image Analyses for COVID-19 Drug Discovery
Simon, M., Schaub, N. J., Yu, S., Ouladi, M., Nagarajan, J., Bayankaram, S. P., Bajcsy, P., & Hotaling, N. A. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00421
Kalliokoski, T. E. A. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669525
“Reviews which focus on the clinical suitability of the models presented like references [3], [4], [5], [6], [7], and [8] have been critical about various aspects of the reviewed papers.”
BackgroundResultInfluentialAnwendungen der künstlichen Intelligenz in der Abdominalbildgebung
Ziegelmayer, S., & Braren, R. (2021) · Gastroenterologie up2date · doi:10.1055/a-1355-0636
“[15] Roberts M, Driggs D, Thorpe M et al.”
BackgroundLaursen, C. B., Prosch, H., Harders, S. W., Falster, C., Davidsen, J. R., & Tárnoki, Á. D. (2021) · European Respiratory Society eBooks · doi:10.1183/2312508x.10012421
Self-citationCombating Bias in COVID-19 Disease Detection Using Synthetic Annotations on Chest X-Ray Images
Ashadi, A. N., Purwita, A. A., & Qomariyah, N. N. (2021) · IEEE International Biomedical Instrumentation and Technology Conference (IBITeC) · doi:10.1109/ibitec53045.2021.9649129
Glüer, C., Krekiehn, N. R., Yilmaz, E. B., & Both, M. (2022) · Osteologie/Osteology · doi:10.1055/a-1717-0955
“Gerade zu COVID-19 gibt es aus den letzten beiden Jahren tausende von Veröffentlichungen zu diesem Thema und praktisch alle Arbeiten sind kritikbedürftig [33], es ist eben noch ein sehr neues Feld und der Veröffentlichungsdruck unter der Pan - demie ging nicht selten auf Kosten von Qualität.”
BackgroundPino, P., Parra, D., Dunstan, J., & Besa, C. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4052411
Demystifying Artificial Intelligence Technology in Cardiothoracic Imaging: The Essentials
Wolterink, J. M., & Mukhopadhyay, A. (2022) · Contemporary medical imaging · doi:10.1007/978-3-030-92087-6_2
Transfer Learning-Based Deep Learning Models for Screening Covid-19 Infection from Chest CT Images
Malliga, S., Kogilavani, S. V., Deepti, R., Krishnan, S. G., & Adhithiya, G. J. (2022) · International Journal of Communications · doi:10.46300/9107.2022.16.7
“However, the current approach has significant flaws in datasets [27], forecasting negative values [26], and removing CAP from COVID-19 [28].”
Validation of a Deep Learning Model to aid in COVID-19 Detection from Digital Chest RadiographsPreprint
Ajmera, P., Kharat, A. T., Khaladkar, S. M., Gupte, T., Pant, R., Kulkarni, V., Duddalwar, V. A., Patkar, D. P., Bhatia, M., & Lamghare, P. K. (2022) · medRxiv · doi:10.1101/2022.06.02.22275895
“In Psychiatry of 2 pandemics 2019 (7-35).”
BackgroundA Shallow Learning Investigation for COVID-19 Classification
Zedda, L., Loddo, A., & Ruberto, C. D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-13321-3_29
Gallego-Díaz, E., Cristancho-Rojas, C., & Críales-Vera, S. (2022) · Archivos de cardiología de México · doi:10.24875/acm.220001481
Automated Classification of Lung Injury from X-ray Images using Deep Learning Network
Le, H., & Do, T. H. (2022) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipaasc55919.2022.9979927
Bárcenas, R. E., & Fuentes–García, R. (2022) · Springer proceedings in mathematics & statistics · doi:10.1007/978-3-031-12778-6_4
AI and Point of Care Image Analysis for COVID-19
Roberts, M. T., Frank, O., Bagon, S., Eldar, Y. C., & Schönlieb, C. (2022) · Springer eBooks · doi:10.1007/978-3-031-08506-2_4
“The systematic reviews most relevant to this Chapter are [13, 17, 14, 15, 12, 16, 8, 11, 18, 19, 20, 21, 22, 23, 24, 28, 29, 30, 31, 32, 33, 34, 35].”
BackgroundInfluentialSelf-citationSimeone, O. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009072205.002
When and How to Use Machine Learning
Simeone, O. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009072205.003
Altaf, F., Islam, S. M. S., Janjua, N. K., & Akhtar, N. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25825-1_9 · Cited by 1
An Advanced Deep Learning Medical Image Recognition and Diagnosis of Respiratory System Viruses
Tayel, M. B., Fahar, A. M. M. E., & Fahmy, A. (2022) · International Conference on Computer and Applications (ICCA) · doi:10.1109/icca56443.2022.10039644
“For the automation of this kind of diagnosis, a computer-aided diagnosis (CAD) system offers important resolutions to research workers in detection of the coronavirus with the help of classification techniques [5, 6].”
Method
Metadata
- Authors
- Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., Weir-McCall, J. R., Teng, Z., Gkrania-Klotsas, E., AIX-COVNET, Ruggiero, A., Korhonen, A., Jefferson, E., Ako, E., Langs, G., Gozaliasl, G., Yang, G., Prosch, H., Preller, J., Stanczuk, J., Tang, J., Hofmanninger, J., Babar, J., Sánchez, L. E., Thillai, M., Gonzalez, P. M., Teare, P., Zhu, X., Patel, M., Cafolla, C., Azadbakht, H., Jacob, J., Lowe, J., Zhang, K., Bradley, K., Wassin, M., Holzer, M., Ji, K., Ortet, M. D., Ai, T., Walton, N., Lio, P., Stranks, S., Shadbahr, T., Lin, W., Zha, Y., Niu, Z., Rudd, J. H. F., Sala, E., & Schönlieb, C.-B.
- Affiliations
- AstraZeneca (United Kingdom) (GB)
- University of Cambridge (GB)
- University of Manchester (GB)
- Royal Papworth Hospital (GB)
- University of Dundee (GB)
- Royal Brompton Hospital (GB)
- Medical University of Vienna (AT)
- University of Helsinki (FI)
- Imperial College London (GB)
- Boehringer Ingelheim (Germany) (DE)
- Addenbrooke's Hospital (GB)
- University College London (GB)
- Macau University of Science and Technology (MO)
- Tongji Hospital (CN)
- Renmin Hospital of Wuhan University (CN)
- Corresponding author
- Roberts, Michael
- ORCID
- Roberts, Michael
- Thorpe, Matthew
- Gilbey, Julian
- Yeung, Michael
- Ursprung, Stephan
- McCague, Cathal
- Weir-McCall, Jonathan R.
- Gkrania-Klotsas, Effrossyni
- Ako, Emmanuel
- Langs, Georg
- Gozaliasl, Ghassem
- Yang, Guang
- Prosch, Helmut
- Preller, Jacobus
- Azadbakht, Hojjat
- Jacob, Joseph
- Ji, Kangyu
- Walton, Nicholas
- Stranks, Samuel
- Shadbahr, Tolou
- Rudd, James H. F.
- Sala, Evis
- Schönlieb, Carola-Bibiane
- Venue
- Nature Machine Intelligence
- Published
- 2021-03-15
- Volume
- 3
- Issue
- 3
- Pages
- 199-217
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W3119527628
- Cited by
- 1,044
- References
- 83
- Citations by year
- 2026: 203 · 2025: 96 · 2024: 143 · 2023: 248 · 2022: 224 · 2021: 130
- FWCI
- 98.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Machine Intelligence (publishedVersion, cc-by)
- Discovery Research Portal (University of Dundee) (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- elib (German Aerospace Center) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Työväentutkimus Vuosikirja (submittedVersion)
- arXiv (Cornell University)
- Funders
- AstraZeneca
- European Commission (777826)
- Medical Research Council (1966157, MR/S010351/1)
- Engineering and Physical Sciences Research Council (EP/T017961/1, EP/N014588/1, EP/S026045/1)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- COVID-19 pandemic, machine learning, chest radiograph, chest CT, COVID-19 detection, outcome prediction, clinical utility, model validation, methodological errors, bias, research recommendations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
