The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.
Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
Open accessAbstract
Abstract Machine learning methods offer great promise for fast and accurate detection and prognostication of coronavirus disease 2019 (COVID-19) from standard-of-care chest radiographs (CXR) and chest computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we consider all published papers and preprints, for the period from 1 January 2020 to 3 October 2020, which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. All manuscripts uploaded to bioRxiv, medRxiv and arXiv along with all entries in EMBASE and MEDLINE in this timeframe are considered. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher-quality model development and well-documented manuscripts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Much of the current evidence is based on retrospective analyses with limited prospective validation [19,28].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The success of deep learning in computer vision (CV) analysis has significantly advanced its clinical applications in comprehending biomedical images, including CT scans [88], positron emission tomography (PET)-CT scans [89], whole-slide images (WSIs) [90], and MRI scans [91].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [88]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Based on this study, we suggest that such a gold rush may give rise to un-engaged 'augmentation' which is highly risky from a learning and knowledge perspective for experts, AI companies, and consumers.”
From To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis (2022) · cites this paper as Roberts et al. 2021
In the citing paper
“Despite problems like the one discovered by Roberts et al. [195], the fact that ML systems process and analyze data differently from human beings can be a great benefit to science and society because these systems may be able to identify useful and innovative structures, properties, patterns, and relationships that human beings would not recognize.”
From The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool (2025) · cites this paper as [195]
In the citing paper
“Differences in image acquisition and processing may result in “shortcuts” where models take cues from non-clinical features.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“For example, COVID-19 prediction models have been demonstrated to learn which institution the images were obtained from rather than features pertaining to the underlying disease. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“The methodological flaws likely is a reflection of the lack of diversity in the development team to allow clinicians, data analysts and patient advocacy groups to work side-by-side throughout the AI life cycle. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“A similar result is summarized in a systematic review of COVID-19 prediction models on CT scans and Chest X-rays (CXRs) that found limited clinical utility due to methodological flaws and underlying biases stemming from small training data sets, data set variability, and limited integration of non-imaging data, among others.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Wang, D., Hu, B., Hu, C., Zhu, F., Liu, X., Zhang, J., Wang, B., Xiang, H., Cheng, Z., Xiong, Y., et al. (2020) · JAMA · doi:10.1001/jama.2020.1585 · Cited by 20,148
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Ai, T., Yang, Z., Hou, H., Zhan, C., Chen, C., Lv, W., Tao, Q., Sun, Z., & Xia, L. (2020) · Radiology · doi:10.1148/radiol.2020200642 · Cited by 5,793
Self-citationRadiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688
Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning
Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C. S., Liang, H., Baxter, S. L., McKeown, A., Yang, G., Wu, X., Yan, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.010 · Cited by 4,842
Self-citation
Research works printed but not linked: 7
- Wong, H. Y. F. et al. Frequency and distribution of chest radiographic findings in COVID-19 positive patients. Radiology 296, 201160 (2019).
- Bassi, P. R. A. S. & Attux, R. A deep convolutional neural network for COVID-19 detection using chest X-rays. Preprint at http://arxiv.org/abs/2005.01578 (2020). · arXiv
- Amyar, A., Modzelewski, R. & Ruan, S. Multi-task deep learning based CT imaging analysis for COVID-19: classification and segmentation. Preprint at medRxiv https://doi.org/10.1101/2020.04.16.2006470 (2020). · DOI
- Goodfellow, I. J. et al. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems Vol. 2 2672–2680 (MIT Press, 2014).
- Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations ICLR 2017 (ICLR, 2017).
- Covidence Systematic Review Software v2486 6284093b (Veritas Health Innovation, 2020); https://www.covidence.org/
- Aviles-Rivero, A. I., Sellars, P., Schönlieb, C.-B. & Papadakis, N. GraphXCOVID: explainable deep graph diffusion pseudo-labelling for identifying COVID-19 on chest X-rays. Preprint at https://arxiv.org/abs/2010.00378 (2020). · arXiv
Cited by
Sorted by citations
A guide to machine learning for biologists
Greener, J. G., Kandathil, S. M., Moffat, L., & Jones, D. T. (2022) · Nature Reviews Molecular Cell Biology · doi:10.1038/s41580-021-00407-0 · Cited by 2,127
Computational approaches streamlining drug discovery
Sadybekov, A., & Katritch, V. (2023) · Nature · doi:10.1038/s41586-023-05905-z · Cited by 1,424
“It has been shown that DL models, especially based on limited datasets lacking negative data, are prone to overtraining and spurious performance, sometimes leading to whole classes of models deemed ‘useless’ 88 or severely biased by subjective factors defining the training dataset 89 .”
BackgroundLeakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
Recent advances and clinical applications of deep learning in medical image analysis
Chen, X., Wang, X., Zhang, K., Fung, K.-M. A., Thai, T., Moore, K. N., Mannel, R. S., Liu, H., Zheng, B., & Qiu, Y. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102444 · Cited by 1,180
Self-citationRedefining Radiology: A Review of Artificial Intelligence Integration in Medical Imaging
Najjar, R. (2023) · Diagnostics · doi:10.3390/diagnostics13172760 · Cited by 1,025
“…mathematical mirroring of reality, depends not only on their training datasets and precision but also on calibration in interpreting medical images depends on comprehensive datasets that accurately reflect varied patient demographics, including age, sex, ethnicity, and disease stages [94,95].”
BackgroundYang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717
Self-citationMachine learning for medical imaging: methodological failures and recommendations for the future
Varoquaux, G. P., & Cheplygina, V. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00592-y · Cited by 815
Lebovitz, S., Lifshitz-Assaf, H., & Levina, N. (2022) · Organization Science · doi:10.1287/orsc.2021.1549 · Cited by 632
“In the past few years, there has been a gold rush to purchasing AI tools, especially in hospitals with fewer resources and lower standards of care (Gkeredakis et al. 2021, Moran 2018, Roberts et al. 2021).”
Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global review
Celi, L. A., Cellini, J., Charpignon, M.-L., Dee, E. C., Dernoncourt, F., Eber, R., Mitchell, W. G., Moukheiber, L., Schirmer, J., Situ, J., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000022 · Cited by 576
“None of the studies performed a proper assessment of model bias [48].”
BackgroundHealth system-scale language models are all-purpose prediction engines
Jiang, L. Y., Liu, X. C., Nejatian, N. P., Nasir-Moin, M., Wang, D., Abidin, A., Eaton, K., Riina, H. A., Laufer, I., Punjabi, P., et al. (2023) · Nature · doi:10.1038/s41586-023-06160-y · Cited by 460
Self-citationKocak, B., Baessler, B., Bakas, S., Cuocolo, R., Fedorov, A., Maier-Hein, L., Mercaldo, N., Müller, H., Orlhac, F., Pinto dos Santos, D., et al. (2023) · Insights into Imaging · doi:10.1186/s13244-023-01415-8 · Cited by 471
Lebovitz, S., Levina, N., & Lifshitz-Assaf, H. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16564 · Cited by 49
Holmes, T. C., Singh, R., Axt, M., Brenner, S., & Barteit, S. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00700-y · Cited by 413
“Given the chatbot ’ s identity was hidden, this raised questions about how AIs should be trained in order to avoid responding inappropriately to user posts 24 .”
BackgroundInfluentialLeveraging artificial intelligence in the fight against infectious diseases
Wong, F. J., de la Fuente‐Núñez, C., & Collins, J. J. (2023) · Science · doi:10.1126/science.adh1114 · Cited by 331
Luo, K., Chen, X., Zheng, H., & Shi, Z. (2022) · Journal of Energy Chemistry · doi:10.1016/j.jechem.2022.06.049 · Cited by 292
Cabitza, F., Campagner, A., Soares, F., García de Guadiana-Romualdo, L., Challa, F., Sulejmani, A., Seghezzi, M., & Carobene, A. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106288 · Cited by 284
“The same reasoning obviously applies, and if possible even exacerbated, in the case of models that rely on less stable tests, such as imaging [50].”
BackgroundMethodInfluentialAI pitfalls and what not to do: mitigating bias in AI
Gichoya, J. W., Thomas, K., Celi, L. A., Safdar, N., Banerjee, I., Banja, J. D., Seyyed-Kalantari, L., Trivedi, H., & Purkayastha, S. (2023) · The British Journal of Radiology · doi:10.1259/bjr.20230023 · Cited by 277
McGenity, C., Clarke, E. L., Jennings, C., Matthews, G., Cartlidge, C., Freduah-Agyemang, H., Stocken, D. D., & Treanor, D. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01106-8 · Cited by 290
“DespitethemanyinterestingdiscoveriesinpathologyAI,translationto routine clinical use remains rare and there are many questions and chal-lengesaroundtheevidencequality,riskofbiasandrobustnessofthemedical AItoolsingeneral 22 – 24, 83 , 84 .”
The need to separate the wheat from the chaff in medical informatics
Cabitza, F., & Campagner, A. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104510 · Cited by 288
Sounderajah, V., Ashrafian, H., Rose, S., Shah, N. H., Ghassemi, M., Golub, R., Kahn, C. E., Esteva, A., Karthikesalingam, A., Mateen, B. A., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01517-0 · Cited by 284
Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging
Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., Tomasev, N., Mitrović, J., Strachan, P., et al. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01049-7 · Cited by 250
Deep learning as a tool for ecology and evolution
Borowiec, M. L., Dikow, R. B., Frandsen, P. B., McKeeken, A. J., Valentini, G., & White, A. E. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13901 · Cited by 295
“…deep learning to ecology and evolution continue to proliferate, we hope for increased scrutiny and mitigation of data biases and other common issues, some of which can be remedied by existing best practices checklists (Roberts et al., 2021) and algorithmic auditing (Juneja and Mitra, 2021).”
BackgroundThe Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251
“Notably, the COVID-19 technology which resulted from innovation policy and democratization efforts mentioned in section 3 was later shown to be completely unsuitable for clinical deployment after the fact [85, 97, 161, 206].”
BackgroundThe promise of digital healthcare technologies
Yeung, A. W. K., Torkamani, A., Butte, A. J., Glicksberg, B. S., Schuller, B. W., Rodríguez, B., Ting, D. S. W., Bates, D. W., Schaden, E., Peng, H., et al. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1196596 · Cited by 275
“, potential inclusion of mislabeled data or data from unknown sources) such that their predictive performances might be over-optimistic (69) or of limited potential clinical use (70).”
Accelerating antibiotic discovery through artificial intelligence
Melo, M. C. R., Maasch, J. R. M. A., & de la Fuente-Nunez, C. (2021) · Communications Biology · doi:10.1038/s42003-021-02586-0 · Cited by 234
“A systematic review of 415 studies on ML-based image analysis for COVID-19 diagnosis found that all publications contained serious methodological fl aws or failed to report key information needed for reproducibility and substantiation of claims, such that not a single model was of clinical use 131 .”
BackgroundTechnology readiness levels for machine learning systems
Lavin, A., Lee, C. M., Visnjic, A., Ganju, S., Newman, D. J., Ganguly, S., Lange, D., Baydin, A. G., Sharma, A., Gibson, A., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33128-9 · Cited by 247
Avoiding common machine learning pitfalls
Lones, M. A. (2024) · Patterns · doi:10.1016/j.patter.2024.101046 · Cited by 298
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204
“This misalignment is analogous to the discrepancies between the recent burst of COVID-related machine learning papers and clinical practices [88].”
ResultA Brief History of AI: How to Prevent Another Winter (A Critical Review)
Toosi, A., Bottino, A., Saboury, B., Siegel, E. L., & Rahmim, A. (2021) · PET Clinics · doi:10.1016/j.cpet.2021.07.001 · Cited by 265
“According to a recent study [97] reviewing a spectrum of machine learning approaches for detecting and prognosticating coronavirus disease 2019 from standard-of-care chest radiographs and computed tomography images, none of the more than 400 studies were found suitable for clinical application!”
BackgroundResnik, D. B., & Hosseini, M. (2025) · AI and Ethics · doi:10.1007/s43681-024-00493-8 · Cited by 228
“Despite problems like the one discovered by Roberts et al. [195], the fact that ML systems process and analyze data differently from human beings can be a great benefit to science and society because these systems may be able to identify useful and innovative structures, properties, patterns, and…”
InfluentialBeker, W., Roszak, R., Wołos, A., Angello, N. H., Rathore, V., Burke, M. D., & Grzybowski, B. A. (2022) · Journal of the American Chemical Society · doi:10.1021/jacs.1c12005 · Cited by 196
Evaluation and Mitigation of Racial Bias in Clinical Machine Learning Models: Scoping Review
Huang, J., Galal, G., Etemadi, M., & Vaidyanathan, M. (2022) · JMIR Medical Informatics · doi:10.2196/36388 · Cited by 186
“Regrettably, medical ML research falls far short of these standards [57,58].”
Brain–phenotype models fail for individuals who defy sample stereotypes
Greene, A. S., Shen, X., Noble, S., Horien, C., Hahn, C. A., Arora, J., Tokoglu, F., Spann, M. N., Carrión, C. I., Barron, D. S., et al. (2022) · Nature · doi:10.1038/s41586-022-05118-w · Cited by 167
Advances in Artificial Intelligence for Infectious-Disease Surveillance
Brownstein, J. S., Rader, B. M., Astley, C. M., & Tian, H. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2119215 · Cited by 195
Bleher, H., & Braun, M. (2022) · AI and Ethics · doi:10.1007/s43681-022-00135-x · Cited by 193
“Systematic reviews, however, indicate that such AI-driven predictive or diagnostic models are not yet appropriate for clinical use due to methodological flaws and their susceptibility to bias [9, 10].”
BackgroundRadiomics, machine learning, and artificial intelligence—what the neuroradiologist needs to know
Wagner, M. W., Namdar, K., Biswas, A., Monah, S. R., Khalvati, F., & Ertl-Wagner, B. B. (2021) · Neuroradiology · doi:10.1007/s00234-021-02813-9 · Cited by 179
Ethical Issues of Digital Twins for Personalized Health Care Service: Preliminary Mapping Study
Huang, P., Kim, K.-H., & Schermer, M. (2021) · Journal of Medical Internet Research · doi:10.2196/33081 · Cited by 158
“Similar problems were also identified in machine learning research that aimed to capitalize on chest radiographs and computed tomography scans to detect COVID-19 [42].”
ResultMaleki, F., Ovens, K. L., Gupta, R., Reinhold, C., Spatz, A., & Forghani, R. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.220028 · Cited by 176
“10 can be utilized responsibly.(34,35) In addition, other current literature offers guidelines for presenting ML and DL research to ensure the reproducibility of the results.”
BackgroundResultA medical multimodal large language model for future pandemics
Liu, F., Zhu, T., Wu, X., Yang, B., You, C., Wang, C., Lu, L., Liu, Z., Zheng, Y., Sun, X., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00952-2 · Cited by 155
“…this manner, Med-MLLM handles visual, textual and multi-modal input, and therefore can be applied to COVID-19 reporting (i.e., medical report generation), diagnosis (i.e., disease classi fi cation), and prognosis (i.e., survival prediction) tasks with limited labels for training 1,2,12,13,15,27,35 .”
InfluentialPuthongkham, P., Wirojsaengthong, S., & Suea‐Ngam, A. (2021) · The Analyst · doi:10.1039/d1an01148k · Cited by 152
REFORMS: Consensus-based Recommendations for Machine-learning-based Science
Kapoor, S., Cantrell, E. M., Peng, K., Pham, T. H., Bail, C. A., Gundersen, O. E., Hofman, J. M., Hullman, J. R., Lones, M. A., Malik, M. M., et al. (2024) · Science Advances · doi:10.1126/sciadv.adk3452 · Cited by 255
Self-citationRavi, V., Acharya, V., & Alazab, M. (2022) · Cluster Computing · doi:10.1007/s10586-022-03664-6 · Cited by 162
“The detailed surveys on lung disease detection are studied and reported in [14, 15, 18, 19].”
Mitigating Bias in Radiology Machine Learning: 1. Data Handling
Rouzrokh, P., Khosravi, B., Faghani, S., Moassefi, M., Vera Garcia, D. V., Singh, Y., Zhang, K., Conte, G. M., & Erickson, B. J. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.210290 · Cited by 145
“…the so-called Frankenstein problem, where one public set is used to train a model and another is used to validate, but the presence of the same case in both sets means the real-world performance is poorer than measured (26). retrieving Digital Imaging and Communications in Medicine (DICOM) files.”
BackgroundSelf-citationHajkowicz, S. A., Sanderson, C., Karimi, S., Bratanova, A., & Naughtin, C. K. (2023) · Technology in Society · doi:10.1016/j.techsoc.2023.102260 · Cited by 186
“An evaluation of 62 published scientific studies using machine learning for COVID-19 diagnosis and prognosis found none of the models could be used for clinical purposes as they were subject to methodological flaws or biases [55].”
MethodImpact of Lung Segmentation on the Diagnosis and Explanation of COVID-19 in Chest X-ray Images
Teixeira, L. O., Pereira, R. M., Bertolini, D. S., Oliveira, L. S., Nanni, L., Cavalcanti, G. D. C., & Costa, Y. M. G. (2021) · Sensors · doi:10.3390/s21217116 · Cited by 152
“Moreover, [19] and [20] presented critical systematic reviews of COVID-19 automatic detection focused on the potential clinical use of the proposed techniques.”
BackgroundUnderstanding and Mitigating Bias in Imaging Artificial Intelligence
Tejani, A. S., Ng, Y. S., Xi, Y., & Rayan, J. C. (2024) · RadioGraphics · doi:10.1148/rg.230067 · Cited by 138
Introduction to radiomics for a clinical audience
McCague, C., Ramlee, S., Reinius, M., SELBY, I. A., Hulse, D., Piyatissa, P., Bura, V., Crispin‐Ortuzar, M., Sala, E., & Woitek, R. (2023) · Clinical Radiology · doi:10.1016/j.crad.2022.08.149 · Cited by 137
Self-citationThe impact of imputation quality on machine learning classifiers for datasets with missing values
Shadbahr, T., Roberts, M. T., Stanczuk, J., Gilbey, J., Teare, P., Dittmer, S., Thorpe, M., Viñas, R., Sala, E., Lió, P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00356-z · Cited by 167
Self-citationRavi, V., Narasimhan, H., Chakraborty, C., & Pham, T. D. (2021) · Multimedia Systems · doi:10.1007/s00530-021-00826-1 · Cited by 144
“Most of the existing techniques to diagnose COVID-19 from CT scans and X-rays have major limitation of dataset used for training or mistakes in methodologies introducing biases [22].”
BackgroundMachine learning applications for COVID-19 outbreak management
Heidari, A., Navimipour, N. J., Ünal, M., & Toumaj, S. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07424-w · Cited by 133
“Misinformation detection has been formalized as a text classification or semantic similarity issue [80].”
BackgroundRecent methodological advances in federated learning for healthcare
Zhang, F., Kreuter, D., Chen, Y., Dittmer, S., Tull, S., Shadbahr, T., Schut, M. C., Asselbergs, F. W., Kar, S., Sivapalaratnam, S., et al. (2024) · Patterns · doi:10.1016/j.patter.2024.101006 · Cited by 152
Self-citationDeep Learning and Medical Image Analysis for COVID-19 Diagnosis and Prediction
Liu, T., Siegel, E. L., & Shen, D. G. (2022) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-110220-012203 · Cited by 135
Don’t push the button! Exploring data leakage risks in machine learning and transfer learning
Apicella, A., Isgrò, F., & Prevete, R. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11326-3 · Cited by 176
Park, S. J., Kim, G., Oh, Y., Seo, J. B., Lee, S. M., Kim, J. H., Moon, S.-J., Lim, J., & Ye, J. C. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102299 · Cited by 127
FDA-regulated AI Algorithms: Trends, Strengths, and Gaps of Validation Studies
Ebrahimian, S., Kalra, M. K., Agarwal, S., Bizzo, B. C., Elkholy, M., Wald, C., Allen, B., & Dreyer, K. J. (2022) · Academic Radiology · doi:10.1016/j.acra.2021.09.002 · Cited by 120
Li, H., Zeng, N., Wu, P., & Clawson, K. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118029 · Cited by 119
McGreivy, N., & Hakim, A. H. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00897-5 · Cited by 172
“Common pitfalls include data leakage [21, 26], poor data quality [21, 24, 27], weak baselines [23], and insufficient external validation [24, 25].”
BackgroundCurrent challenges of implementing artificial intelligence in medical imaging
Saw, S. N., & Ng, K. H. (2022) · Physica Medica · doi:10.1016/j.ejmp.2022.06.003 · Cited by 117
“However, a recent systematic review of a total of 62 AI models that developed to detect COVID-19 using X-ray and CT images have, unfortunately, shown that none could be translated into clinical use due to methodological flaws [6].”
BackgroundHoussein, E. H., Gamal, A. M., Younis, E. M. G., & Mohamed, E. (2025) · Cluster Computing · doi:10.1007/s10586-025-05281-5 · Cited by 148
Is AI leading to a reproducibility crisis in science?
Ball, P. (2023) · Nature · doi:10.1038/d41586-023-03817-6 · Cited by 106
Ly, C. O., Unnikrishnan, B., Tadic, T., Patel, T., Duhamel, J., Kandel, S., Moayedi, Y., Brudno, M., Hope, A. J., Ross, H. J., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01118-4 · Cited by 123
Yadav, P., Menon, N., Ravi, V., & Sowmya, V. (2021) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2021.3103334 · Cited by 107
“According to Roberts et al. [30], none of the current ML literature methods for COVID-19 diagnosis using CT and X-ray imaging is of potential clinical use because of their involved biases and flaws in methodologies.”
MethodAI-Driven Sensing Technology: Review
Chen, L., Xia, C., Zhao, Z., Fu, H., & Chen, Y. (2024) · Sensors · doi:10.3390/s24102958 · Cited by 115
“Reasons for this include biased small datasets, variability in large international dataset sources, and poor integration of multi-stream data, especially imaging datasets [109].”
BackgroundPfob, A., Lu, S.-C., & Sidey-Gibbons, C. J. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01758-8 · Cited by 122
“This reproducibility crisis of medical ML applications has also been observed for models used to diagnose or predict outcomes related to the Covid-19 pandemic and short-term oncologic outcomes [10, 32].”
BackgroundMethodRoth, H. R., Xu, Z., Tor-Díez, C., Jacob, R. S., Zember, J. S., Molto, J., Li, W., Xu, S., Türkbey, B., Turkbey, E. B., et al. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102605 · Cited by 104
Computational drug repurposing: approaches, evaluation of in silico resources and case studies
Tanoli, Z., Fernández‐Torras, A., Özcan, U. O., Kushnir, A., Nader, K. M., Gadiya, Y., Fiorenza, L., Ianevski, A., Vähä-Koskela, M. J. V., Miihkinen, M., et al. (2025) · Nature Reviews Drug Discovery · doi:10.1038/s41573-025-01164-x · Cited by 104
“…on systematic literature reviews, most of the predictive approaches developed to support medical decision-making in COVID-19 suffered from low-quality training data and poor reporting of the model outcomes 194 , and hence the eventual health-care impact of the models remained rather limited 195 .”
Empowering local communities using artificial intelligence
Hsu, Y.-C., Huang, T.-H., Verma, H., Mauri, A., Nourbakhsh, I., & Bozzon, A. (2022) · Patterns · doi:10.1016/j.patter.2022.100449 · Cited by 96
“zens think the AI system can do’’ does not match ‘‘what the AI system can actually do,’’ resulting in socio-technical gaps and pitfalls for actual usage.(37) In our experience, local communities often have high expectations about what AI techniques can do for them—for example, automatically determining whether an indu…”
BackgroundNuclear Medicine and Artificial Intelligence: Best Practices for Algorithm Development
Bradshaw, T., Boellaard, R., Dutta, J., Jha, A. K., Jacobs, P. D., Li, Q., Liu, C., Sitek, A., Saboury, B., Scott, P. J. H., et al. (2021) · Journal of Nuclear Medicine · doi:10.2967/jnumed.121.262567 · Cited by 103
“The hype that propels the development of AI algorithms in nuclear medicine is counterbalanced by concerns about certain pitfalls of AI (1).”
BackgroundHadjiiski, L. M., Cha, K., Chan, H., Drukker, K., Morra, L., Näppi, J. J., Sahiner, B., Yoshida, H., Chen, Q., Deserno, T. M., et al. (2022) · Medical Physics · doi:10.1002/mp.16188 · Cited by 107
Machine learning for polymeric materials: an introduction
Cencer, M. M., Moore, J. S., & Assary, R. S. (2021) · Polymer International · doi:10.1002/pi.6345 · Cited by 97
Blake, N., Gaifulina, R., Griffin, L. D., Bell, I. M., & Thomas, G. M. H. (2022) · Diagnostics · doi:10.3390/diagnostics12061491 · Cited by 103
“This is consistent with other medical domains employing ML [11].”
Passive detection of COVID-19 with wearable sensors and explainable machine learning algorithms
Gadaleta, M., Radin, J. M., Baca-Motes, K., Ramos, E., Kheterpal, V., Topol, E. J., Steinhubl, S. R., & Quer, G. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00533-1 · Cited by 94
Sounds of COVID-19: exploring realistic performance of audio-based digital testing
Han, J., Xia, T., Spathis, D., Bondareva, E., Brown, C., Chauhan, J., Dang, T., Grammenos, A., Hasthanasombat, A., Floto, A., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00553-x · Cited by 90
“Due to these issues, many researchers raised concerns about the feasibility and effectiveness of such models if deployed in real settings [22, 27, 28].”
BackgroundXue, S., Guo, R., Bohn, K. P., Matzke, J., Viscione, M., Alberts, I., Meng, H., Sun, C., Zhang, M., Zhang, M., et al. (2021) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-021-05644-1 · Cited by 90
“A critical concern when using machine learning is its reproducibility and extensibility to unknown complexity in real application [56].”
BackgroundMosquera, C., Ferrer, L., Milone, D. H., Luna, D. R., & Ferrante, E. (2024) · European Radiology · doi:10.1007/s00330-024-10834-0 · Cited by 108
Groen, A. M., Kraan, R., Amirkhan, S. F., Daams, J. G., & Maas, M. (2022) · European Journal of Radiology · doi:10.1016/j.ejrad.2022.110592 · Cited by 89
Wang, X., Barrera, C., Bera, K., Viswanathan, V. S., Azarianpour-Esfahani, S., Koyuncu, C., Velu, P., Feldman, M. D., Yang, M., Fu, P., et al. (2022) · Science Advances · doi:10.1126/sciadv.abn3966 · Cited by 89
“A recent example of the failure of deep learning approaches in real-world clinical settings can be seen in applications for corona-virus disease 2019 (COVID-19) ( 45 ).”
Machine Learning on the COVID-19 Pandemic, Human Mobility and Air Quality: A Review
Rahman, M. M., Paul, K. C., Hossain, M. A., Ali, G. G. M. N., Rahman, M. S., & Thill, J. (2021) · IEEE Access · doi:10.1109/access.2021.3079121 · Cited by 93
“external datasets and external models, and availability and sufficient documentation of public datasets and codes are necessary to develop a trusted and dependable model that can provide consistent accuracy across seen and unseen data and contexts [172]–[175].”
BackgroundDeep Segmentation of the Mandibular Canal: A New 3D Annotated Dataset of CBCT Volumes
Cipriano, M., Allegretti, S., Bolelli, F., Bartolomeo, M. D., Pollastri, F., Pellacani, A., Minafra, P., Anesi, A., & Grana, C. (2022) · IEEE Access · doi:10.1109/access.2022.3144840 · Cited by 85
“We implement their method by applying a 3D connected components labeling (CCL) [36] algorithm on the output masks, preserving only the two largest objects that should represent the left and right branch of the inferior alveolar nerve canal.”
Explainable AI for COVID-19 CT Classifiers: An Initial Comparison Study
Ye, Q., Xia, J., & Yang, G. (2021) · 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS) · doi:10.1109/cbms52027.2021.00103 · Cited by 83
Self-citationArina, P., Kaczorek, M. R., Hofmaenner, D. A., Pisciotta, W., Refinetti, P., Singer, M., Mazomenos, E. B., & Whittle, J. (2024) · Anesthesiology · doi:10.1097/aln.0000000000004764 · Cited by 81
“%) studies applied external validation, 26,27,42,52,58,74,78,80,83,98,112,114,126 while the remaining 87% used only internal validation.”
Gallifant, J., Zhang, J., del Pilar Arias Lopez, M., Zhu, T., Camporota, L., Celi, L. A., & Formenti, F. (2022) · British Journal of Anaesthesia · doi:10.1016/j.bja.2021.09.025 · Cited by 72
Deep GRU-CNN Model for COVID-19 Detection From Chest X-Rays Data
Shah, P. M., Ullah, F., Shah, D., Gani, A., Maple, C. R., Wang, Y., Shahid, S., Abrar, M. Y. B. T., & Islam, S. U. (2021) · IEEE Access · doi:10.1109/access.2021.3077592 · Cited by 82
“Chinese government, the detection of COVID-19 through gene sequencing can be performed using Reverse Transcription Polymerase Chain Reaction (RT-PCR) [12], [13].”
MethodAlami, H., Lehoux, P., Papoutsi, C., Shaw, S. E., Fleet, R. P., & Fortin, J. (2024) · BMC Health Services Research · doi:10.1186/s12913-024-11112-x · Cited by 104
“Roberts et al. (2021) analysed 415 AI models for COVID-19 detection and concluded similarly [25].”
MethodLondon, A. J. (2022) · Cell Reports Medicine · doi:10.1016/j.xcrm.2022.100622 · Cited by 76
“25,39,40,48 The Watson program in health, once described as the ‘‘future of healthcare,’’ has been ‘‘sold for scraps’’ in what some commentators describe as the ‘‘total failure’’ of a program that over-promised and under-delivered from the start.”
Rois, R., Ray, M., Rahman, A., & Sk, R. (2021) · Journal of Health Population and Nutrition · doi:10.1186/s41043-021-00276-5 · Cited by 94
“ML in healthcare generally aims to predict some clinical outcomes on the basis of multiple predictors [29, 30].”
BackgroundComprehensive Review on Facemask Detection Techniques in the Context of Covid-19
Nowrin, A., Afroz, S., Rahman, M. S., Mahmud, I., & Cho, Y.-Z. (2021) · IEEE Access · doi:10.1109/access.2021.3100070 · Cited by 75
“Even, a systematic review article related to diagnosis and prognosis of the Covid-19 using chest radiographs (CXR) and computed tomography (CT) imaging by adopting ML algorithms has been presented in [32].”
BackgroundNikparvar, B., Rahman, M. M., Hatami, F., & Thill, J. (2021) · Scientific Reports · doi:10.1038/s41598-021-01119-3 · Cited by 79
“However, high-quality data devoid of collection biases, robust methodologies, and validation with external data sets and models are necessary to develop a ubiquitous, reliable, and trustworthy model to provide consistent predictions across diverse systems (e.g., data, settlement contexts) 15,43,46 .”
BackgroundStandardized Reporting of Machine Learning Applications in Urology: The STREAM-URO Framework
Kwong, J. C. C., McLoughlin, L. C., Haider, M. A., Goldenberg, M. G., Erdman, L., Rickard, M., Lorenzo, A. J., Hung, A. J., Farcas, M. A., Goldenberg, L., et al. (2021) · European Urology Focus · doi:10.1016/j.euf.2021.07.004 · Cited by 72
Karande, P., Gallagher, B. J., & Han, T. Y.-J. (2022) · Chemistry of Materials · doi:10.1021/acs.chemmater.2c01333 · Cited by 77
Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis
Liu, J., Cen, X., Yi, C., Wang, F.-A., Ding, J., Cheng, J., Wu, Q., Gai, B., Zhou, Y., He, R., et al. (2025) · Genomics, Proteomics & Bioinformatics · doi:10.1093/gpbjnl/qzaf011 · Cited by 75
Wardlaw, J. M., Mair, G., von Kummer, R., Williams, M. C., Li, W., Storkey, A., Trucco, E., Liebeskind, D. S., Farrall, A., Bath, P. M., et al. (2022) · Stroke · doi:10.1161/strokeaha.121.036204 · Cited by 71
“Many current papers describe ML models trained on one small dataset from one hospital, 52,53 insufficient to be useful on the variety seen in clinical practice.”
BackgroundShiri, I., Salimi, Y., Pakbin, M., Hajianfar, G., Avval, A. H., Sanaat, A., Mostafaei, S., Akhavanallaf, A., Saberi, A., Mansouri, Z., et al. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105467 · Cited by 70
X-MIR: EXplainable Medical Image Retrieval
Hu, B., Vasu, B., & Hoogs, A. J. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00161 · Cited by 71
“However, care should be taken when using these datasets, as several biases in the dataset have been found which can impact study conclusions [38, 58, 48].”
BackgroundOn the role of artificial intelligence in medical imaging of COVID-19
Born, J., Beymer, D., Rajan, D., Coy, A., Mukherjee, V. V., Manica, M., Prasanna, P. M., Ballah, D., Guindy, M., Shaham, D., et al. (2021) · Patterns · doi:10.1016/j.patter.2021.100269 · Cited by 73
AANet: Adaptive Attention Network for COVID-19 Detection From Chest X-Ray Images
Lin, Z., He, Z., Xie, S., Wang, X., Tan, J., Lu, J., & Tan, B. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3114747 · Cited by 70
Application and potential of artificial intelligence in neonatal medicine
Kwok, T. C., Henry, C., Saffaran, S., Meeus, M., Bates, D. G., Laere, D. V., Boylan, G. B., Boardman, J. P., & Sharkey, D. (2022) · Seminars in Fetal and Neonatal Medicine · doi:10.1016/j.siny.2022.101346 · Cited by 70
“Some of the common pitfalls are small sample sizes, inappropriate handling of missing values and heterogeneity assessment in different population subsets or healthcare settings [47].”
BackgroundMethodThe regulatory gap in digital health and alternative pathways to bridge it
Iqbal, J. D., & Biller‐Andorno, N. (2022) · Health Policy and Technology · doi:10.1016/j.hlpt.2022.100663 · Cited by 67
Surgical data science – from concepts toward clinical translation
Maier‐Hein, L., Eisenmann, M., Sarıkaya, D., März, K., Collins, T., Malpani, A., Fallert, J., Feußner, H., Giannarou, S., Mascagni, P., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102306 · Cited by 432
Islam, W., Jones, M. A., Faiz, R., Sadeghipour, N., Qiu, Y., & Zheng, B. (2022) · Tomography · doi:10.3390/tomography8050200 · Cited by 82
“As a result, most deep learning models have pitfalls or bias and are not acceptable in clinical practice [40].”
Metadata
- Authors
- Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., Weir-McCall, J. R., Teng, Z., Gkrania-Klotsas, E., AIX-COVNET, Ruggiero, A., Korhonen, A., Jefferson, E., Ako, E., Langs, G., Gozaliasl, G., Yang, G., Prosch, H., Preller, J., Stanczuk, J., Tang, J., Hofmanninger, J., Babar, J., Sánchez, L. E., Thillai, M., Gonzalez, P. M., Teare, P., Zhu, X., Patel, M., Cafolla, C., Azadbakht, H., Jacob, J., Lowe, J., Zhang, K., Bradley, K., Wassin, M., Holzer, M., Ji, K., Ortet, M. D., Ai, T., Walton, N., Lio, P., Stranks, S., Shadbahr, T., Lin, W., Zha, Y., Niu, Z., Rudd, J. H. F., Sala, E., & Schönlieb, C.-B.
- Affiliations
- AstraZeneca (United Kingdom) (GB)
- University of Cambridge (GB)
- University of Manchester (GB)
- Royal Papworth Hospital (GB)
- University of Dundee (GB)
- Royal Brompton Hospital (GB)
- Medical University of Vienna (AT)
- University of Helsinki (FI)
- Imperial College London (GB)
- Boehringer Ingelheim (Germany) (DE)
- Addenbrooke's Hospital (GB)
- University College London (GB)
- Macau University of Science and Technology (MO)
- Tongji Hospital (CN)
- Renmin Hospital of Wuhan University (CN)
- Corresponding author
- Roberts, Michael
- ORCID
- Roberts, Michael
- Thorpe, Matthew
- Gilbey, Julian
- Yeung, Michael
- Ursprung, Stephan
- McCague, Cathal
- Weir-McCall, Jonathan R.
- Gkrania-Klotsas, Effrossyni
- Ako, Emmanuel
- Langs, Georg
- Gozaliasl, Ghassem
- Yang, Guang
- Prosch, Helmut
- Preller, Jacobus
- Azadbakht, Hojjat
- Jacob, Joseph
- Ji, Kangyu
- Walton, Nicholas
- Stranks, Samuel
- Shadbahr, Tolou
- Rudd, James H. F.
- Sala, Evis
- Schönlieb, Carola-Bibiane
- Venue
- Nature Machine Intelligence
- Published
- 2021-03-15
- Volume
- 3
- Issue
- 3
- Pages
- 199-217
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W3119527628
- Cited by
- 1,044
- References
- 83
- Citations by year
- 2026: 203 · 2025: 96 · 2024: 143 · 2023: 248 · 2022: 224 · 2021: 130
- FWCI
- 98.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Machine Intelligence (publishedVersion, cc-by)
- Discovery Research Portal (University of Dundee) (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- elib (German Aerospace Center) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Työväentutkimus Vuosikirja (submittedVersion)
- arXiv (Cornell University)
- Funders
- AstraZeneca
- European Commission (777826)
- Medical Research Council (1966157, MR/S010351/1)
- Engineering and Physical Sciences Research Council (EP/T017961/1, EP/N014588/1, EP/S026045/1)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- COVID-19 pandemic, machine learning, chest radiograph, chest CT, COVID-19 detection, outcome prediction, clinical utility, model validation, methodological errors, bias, research recommendations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
