The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.
Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
Open accessAbstract
Abstract Machine learning methods offer great promise for fast and accurate detection and prognostication of coronavirus disease 2019 (COVID-19) from standard-of-care chest radiographs (CXR) and chest computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we consider all published papers and preprints, for the period from 1 January 2020 to 3 October 2020, which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. All manuscripts uploaded to bioRxiv, medRxiv and arXiv along with all entries in EMBASE and MEDLINE in this timeframe are considered. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher-quality model development and well-documented manuscripts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Much of the current evidence is based on retrospective analyses with limited prospective validation [19,28].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The success of deep learning in computer vision (CV) analysis has significantly advanced its clinical applications in comprehending biomedical images, including CT scans [88], positron emission tomography (PET)-CT scans [89], whole-slide images (WSIs) [90], and MRI scans [91].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [88]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Based on this study, we suggest that such a gold rush may give rise to un-engaged 'augmentation' which is highly risky from a learning and knowledge perspective for experts, AI companies, and consumers.”
From To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis (2022) · cites this paper as Roberts et al. 2021
In the citing paper
“Despite problems like the one discovered by Roberts et al. [195], the fact that ML systems process and analyze data differently from human beings can be a great benefit to science and society because these systems may be able to identify useful and innovative structures, properties, patterns, and relationships that human beings would not recognize.”
From The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool (2025) · cites this paper as [195]
In the citing paper
“Differences in image acquisition and processing may result in “shortcuts” where models take cues from non-clinical features.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“For example, COVID-19 prediction models have been demonstrated to learn which institution the images were obtained from rather than features pertaining to the underlying disease. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“The methodological flaws likely is a reflection of the lack of diversity in the development team to allow clinicians, data analysts and patient advocacy groups to work side-by-side throughout the AI life cycle. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“A similar result is summarized in a systematic review of COVID-19 prediction models on CT scans and Chest X-rays (CXRs) that found limited clinical utility due to methodological flaws and underlying biases stemming from small training data sets, data set variability, and limited integration of non-imaging data, among others.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Wang, D., Hu, B., Hu, C., Zhu, F., Liu, X., Zhang, J., Wang, B., Xiang, H., Cheng, Z., Xiong, Y., et al. (2020) · JAMA · doi:10.1001/jama.2020.1585 · Cited by 20,148
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Ai, T., Yang, Z., Hou, H., Zhan, C., Chen, C., Lv, W., Tao, Q., Sun, Z., & Xia, L. (2020) · Radiology · doi:10.1148/radiol.2020200642 · Cited by 5,793
Self-citationRadiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688
Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning
Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C. S., Liang, H., Baxter, S. L., McKeown, A., Yang, G., Wu, X., Yan, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.010 · Cited by 4,842
Self-citation
Research works printed but not linked: 7
- Wong, H. Y. F. et al. Frequency and distribution of chest radiographic findings in COVID-19 positive patients. Radiology 296, 201160 (2019).
- Bassi, P. R. A. S. & Attux, R. A deep convolutional neural network for COVID-19 detection using chest X-rays. Preprint at http://arxiv.org/abs/2005.01578 (2020). · arXiv
- Amyar, A., Modzelewski, R. & Ruan, S. Multi-task deep learning based CT imaging analysis for COVID-19: classification and segmentation. Preprint at medRxiv https://doi.org/10.1101/2020.04.16.2006470 (2020). · DOI
- Goodfellow, I. J. et al. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems Vol. 2 2672–2680 (MIT Press, 2014).
- Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations ICLR 2017 (ICLR, 2017).
- Covidence Systematic Review Software v2486 6284093b (Veritas Health Innovation, 2020); https://www.covidence.org/
- Aviles-Rivero, A. I., Sellars, P., Schönlieb, C.-B. & Papadakis, N. GraphXCOVID: explainable deep graph diffusion pseudo-labelling for identifying COVID-19 on chest X-rays. Preprint at https://arxiv.org/abs/2010.00378 (2020). · arXiv
Cited by
Sorted by citations
Items 601–700 of 1,000
Artificial Intelligence in Biomedicine: Systematic ReviewPreprint
Gabashvili, I. S. (2023) · medRxiv · doi:10.1101/2023.07.23.23292672 · Cited by 4
“A fourth study [25] reinforced the need for high-quality datasets, reproducible studies with thorough documentation, and external validation to increase the likelihood of these models being integrated into future clinical trials.”
BackgroundInfluentialA pipeline to further enhance quality, integrity and reusability of the NCCID clinical data
Breger, A., SELBY, I. A., Roberts, M. T., Babar, J. L., Gkrania‐Klotsas, E., Preller, J., Sánchez, L. E., Dittmer, S., Thorpe, M., Gilbey, J., et al. (2023) · Scientific Data · doi:10.1038/s41597-023-02340-7 · Cited by 4
“A certain level of bias is unavoidable in any dataset; however, its impact may be mitigated by gaining a thorough understanding of the data and any potential sources of bias 3,4 .”
BackgroundSelf-citationFrom Medieval Scapegoating to Modern Conspiracy Theories in HealthcarePreprint
Toma, M. (2025) · Qeios · doi:10.32388/o52hwm.2 · Cited by 3
Ezzeddine, H., Awad, M., Ghanem, A. S. A., & Mourani, B. (2021) · 2021 IEEE 3rd International Multidisciplinary Conference on Engineering Technology (IMCET) · doi:10.1109/imcet53404.2021.9665574 · Cited by 2
Li, J., Zhong, D., Wang, S., Tao, C., & Wang, H. (2025) · IEEE Access · doi:10.1109/access.2025.3544439 · Cited by 2
“A review of AI-based medical imaging solutions highlights ongoing challenges, including data quality issues, suboptimal machine learning applications, and concerns about reproducibility and bias [7].”
Development of tiliqua species provenance models for use in combating the illegal wildlife trade
Brandis, K., Zawada, K. J. A., Meagher, P. J. B., Ramp, D., & Francis, R. J. (2025) · Frontiers in Ecology and Evolution · doi:10.3389/fevo.2025.1526584 · Cited by 2
Kılıç, Ş. (2025) · Çukurova Üniversitesi Mühendislik Fakültesi Dergisi · doi:10.21605/cukurovaumfd.1653486 · Cited by 2
Bielick, C. G., Awwad, A., Ellen, J., Jalilian, L., McCoy, L. G., Mishra, V., Osmanlliu, E., Pfohl, S. R., & Celi, L. A. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001115 · Cited by 2
“The COVID-19 pandemic exposed this reality; one systematic review of 62 published AI models found none fit for clinical use, largely due to training-test distribution mismatches and absent prospective validation [11].”
Iftode, L., Danceanu, C. M. Z., Cumpata, A. J., Popa, M., Labusca, L. S., & Radulescu, L. M. (2025) · BioMedical Engineering OnLine · doi:10.1186/s12938-025-01467-5 · Cited by 2
“Transparent reporting of training pipelines, hyperparameters, and code availability further streamline reproduci - bility [75].”
Reflexivity and positionality statements: a way to tackle “second-level arbitrariness” bias in AI?
Conti, L. G., & Seele, P. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-06208-6 · Cited by 3
“…training data sets, data set variability, and limited integration of non-imaging data ” (Gichoya et al., 2023), were deemed un fi t for clinical use (Roberts et al., 2021), to racial biases in an “ algorithm used to manage the health of populations ” (Obermeyer et al., 2019), or even algorithms…”
Sousa, G. J. A., Sá, R. A., Gomes, M. A. S. M., Edwards, G. A., Moreno‐González, I., & de Sousa, R. J. A. (2026) · Neuroinformatics · doi:10.1007/s12021-025-09763-0 · Cited by 2
“2.1), the images were treated as independent fields (Roberts et al., 2021), and the evaluation was limited to internal validation.”
InfluentialUnderstanding Dataset Bias in Medical Imaging: A Case Study on Chest X-rays
Dack, E., & Dai, C. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00066 · Cited by 4
“Although there were many research publications with papers often achieving over 99% accuracy, these models were more than often deemed unusable in clinical practice [44].”
Survey on mathematical modeling of infectious disease dynamics: insights and applications
Eshtewy, N. A., Forootani, A., Sisi, Z. A., & Forootani, A. (2026) · BMC Infectious Diseases · doi:10.1186/s12879-026-12905-7 · Cited by 2
“…outbreak windows or affected by reporting delays and under-ascer-tainment, as well as data leakage , where temporal or spatial information inadvertently enters training pipelines and leads to overly optimistic performance estimates that fail to generalize to real-time forecasting scenarios [414].”
Whisnant, A. W., & Dölken, L. (2026) · AI · doi:10.3390/ai7030093 · Cited by 3
Voigtländer, H., Schmitz, F., Strauss, D. S., Kauczor, H. ‐., Voigtlander, S., Sauerwein, S., & Sedaghat, S. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2026.1674509 · Cited by 2
Alahmari, M. A. (2026) · Journal of Asthma and Allergy · doi:10.2147/jaa.s575312 · Cited by 2
“Finally, Roberts et al 53 addresses the challenges in applying machine learning to predict lung function decline, highlighting the complexities of developing accurate predictive models from longitudinal human studies.”
Sadeghsalehi, H. (2026) · Array · doi:10.1016/j.array.2026.100778 · Cited by 4
Clinician-accessible automated machine learning in oral healthcare: A systematic review
Shujaat, S., Riaz, M., Almutairi, H., Alanazi, H., Altalhi, L., Alenazi, N., Osayl, H. B., Philip, M. R., Aboalela, A. A., & Ma, H. (2026) · Japanese Dental Science Review · doi:10.1016/j.jdsr.2026.03.001 · Cited by 2
“This mirrors broader findings that most published clinical AI models do not progress to prospective validation or implementation [78,79].”
Nguyen, H. T. T., Le, V. Q., Dinh‐Xuan, A. T., & Nguyen, L. N. (2026) · Diagnostics · doi:10.3390/diagnostics16081127 · Cited by 3
“Vendor developed datasets may contain higher proportions of radiographically obvious disease or images acquired under optimized imaging conditions, which can inflate reported diagnostic performance [16,31].”
Decomposing performance inflation in radiomics: factorial quantification of analytical leakage
García-Hidalgo, C., Hernández, J. A. C., Espí, J. V. C., Vicente, G. P., Ponce, J. F. M., Olmos, A. V., & Martínez, J. M. P. (2026) · European Journal of Radiology · doi:10.1016/j.ejrad.2026.112890 · Cited by 2
Houssein, E. H., Dirar, A., Ibrahim, I. A., & Al-Sayed, M. M. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10628-1 · Cited by 2
“Models trained on such imbalanced, multi-source data frequently overfit and report inflated performance that collapses under external validation [85, 86].”
Research integrity and data ethics in AI-driven integrated healthcare: a critical appraisal
Zeng, W., Wei, H., Chen, Y., Tang, J., Ge, J., & Chen, H. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1838551 · Cited by 2
“Systematic reviews reveal that many published AI studies provide insufficient methodological detail for replication (37, 38).”
Self-citationOberascher, M., Brentan, B., Menapace, A., Herrera, M., Fu, G., Taormina, R., & Sitzenfrei, R. (2026) · Water Research X · doi:10.1016/j.wroa.2026.100569 · Cited by 2
Sadozai, S. K., & Soylak, M. (2026) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2026.119029 · Cited by 2
Palermi, S., Zeidaabadi, B., Vecchiato, M., Anselmino, M., Adorisio, R., Biffi, A., Borrelli, F., Brugin, E., Cantarutti, N., Cavarretta, E., et al. (2026) · International Journal of Cardiology · doi:10.1016/j.ijcard.2026.134677 · Cited by 2
Pozzi, J., D’Argenzio, J., Carriero, S., Cè, M., D’Arrigo, D., Lanza, C., Biondetti, P., Angileri, S. A., Pavan, M., Catona, R., et al. (2026) · Diagnostics · doi:10.3390/diagnostics16142266 · Cited by 2
Sarı, F., & Aliyeva, A. (2026) · Children · doi:10.3390/children13091166 · Cited by 2
“Ghassemi et al. cautioned that model explanations and ranked feature importance can be misleading when interpreted as mechanisms, while DECIDE-AI recommends evaluating explanations within the intended clinical workflow rather than presenting them as causal evidence [25–28].”
Evaluation of Convolutional Neural Networks for COVID-19 Detection from Chest X-Ray Images
Condaragiu, S., & Ciocoiu, I. B. (2021) · International Symposium on Signals, Circuits and Systems · doi:10.1109/isscs52333.2021.9497418 · Cited by 2
Chaotic Flower Pollination with Deep Learning Based COVID-19 Classification Model
Gopalakrishnan, T., Sikkandar, M. Y., Alharbi, R., Selvaraj, P., Kareem, Z. H., Alkhayyat, A. H., & Abbas, A. H. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.033252 · Cited by 2
Berenbaum, A., Delingette, H., Maire, A., Poret, C., Hassen‐Khodja, C., Bréant, S., Daniel, C., Martel, P., Grimaldi‐Bensouda, L., Frank, M., et al. (2023) · Applied Sciences · doi:10.3390/app13095281 · Cited by 2
Freire, N. D. S., Leão, P. P. D. S., Tiago, L. A., Gonçalves, A., Pinto, R. A., Santos, E. M. D., & Souto, E. (2023) · Computer Methods in Biomechanics and Biomedical Engineering Imaging & Visualization · doi:10.1080/21681163.2023.2264408 · Cited by 3
“Unfortunately, this highly imbalanced dataset exhibits a common pattern found in many public datasets used in the literature, with a limited number of instances of patients with COVID-19 (Roberts et al. 2021).”
BackgroundRNN-based Model for an Optimal COVID-19 Cases Detection using Clinical Reports
Khanday, A. M. U. D., Bouktif, S., & Ouni, A. (2023) · International Conference on Optimization and Applications (ICOA) · doi:10.1109/icoa58279.2023.10308812 · Cited by 3
“Roberts et al. [15] inspected research that utilizes AI to identify and guess for Coronavirus from chest radiographs and CT filters.”
BackgroundACRE: Accelerating Random Forests for Explainability
McCrabb, A., Ahmed, A., & Bertacco, V. (2023) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3613424.3623788 · Cited by 2
Retinale optische Kohärenztomographie Biomarker und ihr Zusammenhang mit kognitiven Funktionen
Rauscher, F. G., & Bernardes, R. (2024) · Die Ophthalmologie · doi:10.1007/s00347-024-01985-y · Cited by 2
Retinal OCT biomarkers and their association with cognitive function—clinical and AI approaches
Rauscher, F. G., & Bernardes, R. (2024) · Die Ophthalmologie · doi:10.1007/s00347-024-01988-9 · Cited by 2
Multi-centre benchmarking of deep learning models for COVID-19 detection in chest x-rays
Harkness, R., Frangi, A. F., Zucker, K., & Ravikumar, N. (2024) · Frontiers in Radiology · doi:10.3389/fradi.2024.1386906 · Cited by 2
A High-Performance CNN Hardware Accelerator for Covid-19 Feature Extraction from X-ray Images
Nguyen, H. K., Duong, H. K. L., & Trang, D.-T. (2024) · International Conference on Integrated Circuits, Design, and Verification (ICDV) · doi:10.1109/icdv61346.2024.10616751 · Cited by 2
“According to the World Health Organization (WHO), the diagnosis of COVID-19 through genetic sequencing can be achieved using the Reverse Transcription Polymerase Chain Reaction (RT-PCR) [2], [3].”
MethodAdverserial network augmentation and tabular data for a new covid-19 diagnostics approach
Al-Bwana, E. K., Sayahi, I., Alauthman, M., & Mahjoub, M. A. (2024) · International Conference on Control, Decision and Information Technologies (CoDIT) · doi:10.1109/codit62066.2024.10708515 · Cited by 2
“However, developing reliable AI systems for these tasks is constrained by the limited availability of real-world COVID-19 training data [2].”
BackgroundLepoittevin, M., Blancart‐Remaury, Q., Lévêque, N., Thille, A. W., Brunet, T., Salaun, K., Catroux, M., Pellerin, L., Hauet, T., & Thuillier, R. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms252212199 · Cited by 2
Rural Tourist Attractions Recommendation Model Based on Multi-Feature Fusion Graph Neural Networks
Zhang, X., & Wang, X. (2024) · International Journal of Computational Intelligence and Applications · doi:10.1142/s1469026824500275 · Cited by 2
Costa, M. V. L., de Aguiar, E. J., Rodrigues, L. S., Traina, C., & Traina, A. J. M. (2024) · Health Information Science and Systems · doi:10.1007/s13755-024-00330-6 · Cited by 2
“These models are well-known texture extractors that are well-fitting for medical images [38].”
Specialty grand challenge in neuroinfectious diseases
Misra, U. K. (2025) · Frontiers in Neurology · doi:10.3389/fneur.2025.1557610 · Cited by 2
CoviSegNet - Covid-19 Disease Area Segmentation using Machine Learning Analyses for Lung Imaging
Mittal, B., & Oh, J. (2021) · International symposium on image and signal processing and analysis/ISPA ... · doi:10.1109/ispa52656.2021.9552078 · Cited by 2
Wright, L. S. (2023) · Cambridge University Press eBooks · doi:10.1017/9781009052023.006 · Cited by 2
Chang, K.-P., Lin, S.-H., & Chu, Y.-W. (2021) · Artificial Intelligence in Gastroenterology · doi:10.35712/aig.v2.i2.27 · Cited by 2
Screening of Viral Pneumonia and COVID-19 in Chest X-ray using Classical Machine Learning
Fonseca, A. U., Vieira, G. D. S., & Soares, F. A. A. D. M. N. (2021) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac51774.2021.00294 · Cited by 2
“First, as listed by [4, 5] most of the published work so far uses data sets with little or none study on bias, and even using a more up-to-date and broader version of a data set.”
BackgroundMethodTushar, F. I., Abadi, E., Sotoudeh‐Paima, S., Fricks, R. B., Mazurowski, M. A., Segars, W. P., Samei, E., & Lo, J. Y.-C. (2022) · SPIE Medical Imaging: Computer-Aided Diagnosis · doi:10.1117/12.2613010 · Cited by 2
“These concerns greatly reduce the generalizability of most AI research studies, making them impractical for clinical translation [10, 11].”
BackgroundThe Industrialization of Intelligence
Byczkowski, M., & Görtz, M. (2022) · Springer eBooks · doi:10.1007/978-3-031-04198-3_17 · Cited by 2
New patch-based strategy for COVID-19 automatic identification using chest x-ray images
Portal-Díaz, J. A., Enríquez, O. A. L., Pérez-Díaz, M., López-Cabrera, J. D., Reyes-Cardoso, O., & Orozco‐Morales, R. (2022) · Health and Technology · doi:10.1007/s12553-022-00704-4 · Cited by 3
“In other words, these proposed models suffer from the inherent drawbacks of low generalization capability, derived from the sparse labeled COVID-19 data [22].”
BackgroundResiliente und robuste KI-Systeme im praktischen Einsatz
Wittenbrink, N., Demirci, S., & Wischmann, S. (2022) · Springer eBooks · doi:10.1007/978-3-662-66057-7_12 · Cited by 2
Artificial intelligence in COVID-19
Nessiem, M. A., Coppock, H., Mohamed, M. M., & Schuller, B. W. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-91794-0.00016-0 · Cited by 2
Gasulla, Ó., Ledesma‐Carbayo, M. J., Borrell, L. N., Fortuny-Profitós, J., Mazaira‐Font, F. A., Allende, J. M. B., Alonso-Menchén, D., Bennett, J. G., del Rio, B., Jofré-Grimaldo, H., et al. (2023) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2023.1010679 · Cited by 2
“Thus, we performed a novel analysis method that segmented only lungs and refine the predictions with the convex hull and the ideal mask transformation (Arun et al., 2020; Chen et al., 2020b;HuYuan andZhangming, 2020;Wu et al., 2020; Zheng et al., 2020; Zhu et al., 2020; Ebrahimian et al., 2021; Mushtaq et al., 2021; R…”
MethodMore Than Just a Policy: Day-to-Day Effects of Data Governance on the Data Scientist
Marivate, V. N. (2023) · Springer eBooks · doi:10.1007/978-3-031-24498-8_7 · Cited by 2
Bridge, J., Meng, Y., Zhu, W., FitzMaurice, T. S., McCann, C. A., Addison, C. A., Wang, M., Merritt, C., Franks, S., Mackey, M., et al. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1113030 · Cited by 2
“(16) identified three common areas in which models often fail these are: (1) a lack of adequately documented methods for reproducibility, (2) failure to follow established guidelines and best practices for the development of deep learning models, and (3) an absence of external validation displaying the model’s applica…”
Adedinsewo, D. A., Dugan, J. L., Johnson, P. W., Douglass, E. J., Morales-Lara, A. C., Parkulo, M. A., Ting, H. H., Cooper, L. T., Scott, L. R., Valverde, A. M., et al. (2023) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2023.07.007 · Cited by 2
3. Observation and Measurement
Ingram, D. (2023) · Open Book Publishers · doi:10.11647/obp.0335.03 · Cited by 2
Beyond Model Accuracy: Identifying Hidden Underlying Issues in Chest X-ray Classification
Wainwright, R., Wang, D., Layton, H., & Bialkowski, A. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8388-9_43 · Cited by 2
Neural Network Based Mortality Prediction in Covid-19 Dataset
Rane, R., Dubey, A., Rasool, A., & Wadhvani, R. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-48774-3_25 · Cited by 2
CT quantification of COVID-19 pneumonia extent to predict individualized outcome
Bérecová, Z., Juskanič, D., Hazlinger, M., UHNAK, M., Janega, P., Rudnay, M., & Hatala, R. (2024) · Bratislavské lekárske listy/Bratislava medical journal · doi:10.4149/bll_2024_25 · Cited by 2
Evaluation of Explainable AI Methods in CNN Classifiers of COVID-19 CT Images
Lima, J. P. D. O., d’Amore, R., Máximo, M. R. O. A., Júnior, M. H. V., & Matsumoto, M. M. S. (2024) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-031-49404-8_31 · Cited by 2
Impact of Aligning Saliency Maps on COVID-19 Disease Detection Using Chest X-Ray Images
Purwita, A. A., & Qomariyah, N. N. (2021) · International Seminar on Intelligent Technology and Its Applications · doi:10.1109/isitia52817.2021.9502264 · Cited by 2
Ackall, G., Elmzoudi, M., Yuan, R., & Chen, C. (2021) · Technologies · doi:10.3390/technologies9040098 · Cited by 2
“This process should not be considered as a replacement for RT-PCR swab tests or manual radiologist diagnosis, but instead as a complement [35].”
BackgroundDeveloping interactive computer-aided detection tools to support translational clinical research
Danala, G., Mirniaharikandehei, S., Jones, M. A., Gai, T., Maryada, S. K., Wu, D. H., Qiu, Y., & Zheng, B. (2022) · SPIE Medical Imaging: Image Perception, Observer Performance, and Technology Assessment · doi:10.1117/12.2607273 · Cited by 2
“, small and unnormalized image datasets used in developing deep learning models, and the model training or validation bias as identified and reported by the researchers [1]).”
BackgroundArtificial Neural Networks and BPPC Features for Detecting COVID-19 and Severity Level
Fonseca, A. U., Félix, J. P., Vieira, G. D. S., Fernandes, D. S. A., & Soares, F. A. A. D. M. N. (2022) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53654.2022.9945221 · Cited by 2
“In a much smaller number, some studies use private data ‡ Fabrizzio Soares: Corresponding author (fabrizzio@ufg.br) with a specific cohort and protocol, resulting in difficulties to check by third parties [3]–[7].”
Explaining a XX century horse behaviour
Gozzi, N., & Chiti, A. (2021) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-021-05417-w · Cited by 2
The Importance of Data Quality in the Nascent Algorithmic Age of Radiology
Chepelev, L. L. (2022) · Academic Radiology · doi:10.1016/j.acra.2022.03.003 · Cited by 2
Bernaola, N., de Lima, G., Riaño, M., Llanos, L., Heili-Frades, S. B., Sanchez, O., Lara, A., Plaza, G., Carballo, C., Gallego, P., et al. (2022) · medRxiv · doi:10.1101/2022.05.09.22274832 · Cited by 2
Scantlebury, J., Vost, L., Carbery, A., Hadfield, T. E., Turnbull, O. M., Brown, N., Chenthamarakshan, V., Das, P., Grosjean, H., von Delft, F., et al. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.10.28.511712 · Cited by 2
Shrestha, P., Lohse, H., Bhatla, C., McCartney, H., Alzaki, A. A., Sandhu, N., Oli, P. K., Li, H., Prakash, M., Amid, A., et al. (2024) · medRxiv · doi:10.1101/2024.09.21.24314128 · Cited by 2
The Perks and Perils of Machine Learning in Business and Economic ResearchPreprint
Dudda, T. L., & Hornuf, L. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4981802 · Cited by 2
From Medieval Scapegoating to Modern Conspiracy Theories in HealthcarePreprint
Toma, M. (2025) · Qeios · doi:10.32388/o52hwm · Cited by 2
Benani, A., Ohayon, S., Laleye, F., Bauvin, P., Messas, E., Bodard, S., & Tannier, X. (2025) · medRxiv · doi:10.1101/2025.03.12.25322656 · Cited by 2
Kranz, D. D., Kahriman, O., Dischl, D., Treskatsch, S., Sander, A., Brachmann, J., Park, J.-W., & Wessel, N. (2025) · doi:10.64898/2025.11.30.25341301 · Cited by 1
Ho, A. (2023) · Oxford University Press eBooks · doi:10.1093/med/9780197556269.003.0005 · Cited by 1
Artificial Intelligence, Bioinformatics, and Pathology
Levy, J., Lu, Y., Montivero, M. K. S., Ramwala, O. A., McFadden, J. R., Miles, C., Diamond, A. G., Reddy, R., Reddy, R., Hudson, T. R., et al. (2022) · Advances in Molecular Pathology · doi:10.1016/j.yamp.2023.01.002 · Cited by 1
Mathematics of biomedical imaging today—a perspective
Betcke, M. M., & Schönlieb, C. (2023) · Progress in Biomedical Engineering · doi:10.1088/2516-1091/acd973 · Cited by 1
Self-citationMultimodal Diagnosis of COVID-19 Using Deep Wavelet Scattering Networks
Zewail, R. (2023) · Internet of things · doi:10.1007/978-3-031-28631-5_10 · Cited by 1
Errors and fallibility in radiology: X-ray readings and expert radiologists, 1947–1960
Lysen, F. (2023) · BJHS Themes · doi:10.1017/bjt.2023.5 · Cited by 1
Lippolis, E. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-13681-8.00015-1 · Cited by 1
Distribution-based detection of radiographic changes in pneumonia patterns: A COVID-19 case study
Pereira, S. C., Rocha, J., Campilho, A., & Mendonça, A. M. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e35677 · Cited by 1
“Issues such as shortcut learning and dataset shifts, often due to each data class being collected from different sources, were key factors behind apparent successes in COVID-19 classification algorithms [11,32].”
Automation to approximate the Gestalt: applying machine learning to the general movement assessment
Garfinkle, J. (2024) · Pediatric Research · doi:10.1038/s41390-024-03558-w · Cited by 2
“16 A useful algorithm for prognostication must be robust enough to give accurate results for any sample of infants rather than just on the initially sampled cohort.”
Santos, F., Zanchettin, C., Lei, W., & Amaral, L. A. N. (2024) · Physical review. E · doi:10.1103/physreve.110.054310 · Cited by 2
“In shortcut learning, the models learn irrelevant (to humans and to the general task) but effective artifacts in the training dataset [12,13,51].”
Importance of the Data in the Surgical Environment
Rivoir, D., Wagner, M., Bodenstedt, S., März, K., Kolbinger, F. R., Maier-Hein, L., Seidlitz, S., Brandenburg, J. M., Müller-Stich, B. P., Distler, M., et al. (2024) · Springer eBooks · doi:10.1007/978-3-031-68574-3_2 · Cited by 1
Drukker, K., Armato, S. G., Hadjiiski, L. M., Gichoya, J. W., Gruszauskas, N., Kalpathy–Cramer, J., Li, H., Myers, K. J., Tomek, R. M., Whitney, H. M., et al. (2025) · SPIE Medical Imaging: Image Perception, Observer Performance, and Technology Assessment · doi:10.1117/12.3045457 · Cited by 1
Lightweight Concurrent Out-of-Distribution Detection in Hyperdimensional Computing Hardware
Roodsari, M. S., Meyers, V., & Tahoori, M. B. (2025) · IEEE International On-Line Testing Symposium (IOLTS) · doi:10.1109/iolts65288.2025.11116825 · Cited by 1
“Low-latency and high-accuracy OOD detection is essential for the reliability of ML models, especially in safety-critical scenarios, such as medical diagnostic system [8] or autonomous driving [6].”
Williamson, A., Tavabi, N., Kalepalli, N., Lavoie-Gagne, O. Z., Weiss, A., Bijwadia, S. R., Sibley, A., Owens, B., Andujar, R. A. G., Singh, H., et al. (2026) · Scientific Reports · doi:10.1038/s41598-026-52928-3 · Cited by 1
“Additionally, GPT-o4-mini outperforms the admin set by 0.09 in micro-averaged F1-score and 0.10 in macro-averaged F1-score.”
InfluentialPokhrel, L., Kumar, A., Garg, P., Anand, N., & Singh, N. S. S. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-4134-7.ch013 · Cited by 3
ESCMID workshop: Artificial intelligence and machine learning in medical microbiology diagnostics
Greutmann, M., Borgwardt, K. M., Brüningk, S. C., Franzeck, F. C., Giske, C. G., Green, A. G., Guerrero-López, A., Ip, M., Jutzeler, C. R., Kahles, A., et al. (2025) · Microbes and Infection · doi:10.1016/j.micinf.2025.105562 · Cited by 1
Kumar, S., Malviya, R., Sridhar, S. B., Shareef, J., & Wadhwa, T. (2025) · Anti-Inflammatory & Anti-Allergy Agents in Medicinal Chemistry · doi:10.2174/0118715230389406250906142050 · Cited by 1
DRGM: Data Requirement Goal Modeling for ML-Based Systems
Yamani, A. Z., AlAmoudi, N., Albelali, S., Baslyman, M., & Hassine, J. (2025) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew66121.2025.00013 · Cited by 3
Fund, F., Saeed, M., Malik, S., & Ishak, K. (2025) · ACM Conference on Reproducibility and Replicability (ACM REP) · doi:10.1145/3736731.3746153 · Cited by 1
“…chest X-Ray images using deep learning: A comparison of transfer learning approaches [16]: This paper is one of many in the academic literature that trains a machine learning model to classify COVID-19 from chest X rays, but the model has no clinical value because of a data leakage error [7, 18].”
Salma, S. U., Renduchintala, C. R., Siddique, I., Sterling, E. W., Sneha, S., & Sakib, N. (2025) · JMIR Mental Health · doi:10.2196/79973 · Cited by 1
“These gaps make it hard to see important methodological choices about data cleaning, normalization, and integration that can have a significant impact on AI performance [79,80].”
CIRCA: comprehensible online system in support of chest X-rays-based screening by COVID-19 example
Prazuch, W., Suwalska, A., Socha, M., Tobiasz, J., Foszner, P., Jaroszewicz, J., Gruszczyńska, K., Sliwinska, M., Nowak, M., Giżycka, B., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-26895-0 · Cited by 1
“Multiple deep learning models were developed to predict COVID-19 based on CXR images 11,12,17 .”
Artificial Intelligence Research as a Continuous Clinical Service
Xu, Y., Dodhia, R., Ferres, J. M. L., & Weeks, W. B. (2025) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2025.100330 · Cited by 1
“19 Although insights from the medical literature can identify spurious correlations, metic-ulous data exploration before modeling, including analyzing correlations between clinical outcomes and clinically irrelevant features, can expose latent confounders.”
Hansen, W., Munir, B., & Jarvis, E. E. (2026) · SN Comprehensive Clinical Medicine · doi:10.1007/s42399-026-02253-7 · Cited by 1
“…hospital systems and Picture Archiving and Communica - tion Systems (PACS) can also be technically complex, and models trained in high-resource centres may perform less well in rural hospitals or LMICs, where imaging infrastruc - ture, disease patterns, and workflow constraints differ [24 , 25].”
Diagnostic Performance of Artificial Intelligence in Detecting COVID-19 Pneumonia on Chest Imaging
Chapman, S. R., Willner, L., Abouafech, A., Roberti, C., & Willner, C. (2026) · Cureus · doi:10.7759/cureus.101775 · Cited by 2
Understanding Artificial Intelligence (AI) for the Electrophysiologist
Ramanathan, C., & Trayanova, N. A. (2026) · Indian Pacing and Electrophysiology Journal · doi:10.1016/j.ipej.2026.01.010 · Cited by 1
“Several high-profile evaluations of medical AI have shown large drops in performance when internally developed models were applied prospectively or in external settings, underscoring the gap between promising prototypes and deployable clinical tools [22,23].”
InfluentialKomninos, P., Kontogiannis, T., Eleftheroglou, N., & Zarouchas, D. S. (2026) · Data-Centric Engineering · doi:10.1017/dce.2025.10031 · Cited by 1
“On the first hand, the complexity and heterogeneity of medical datasets pose a significant challenge in extracting actionable knowledge from data (Roberts et al., 2021; Sapoval et al., 2022).”
Machine-learning algorithms for detecting intracranial hemorrhage on head computed tomography
Goelz, L., Stengel, D., Käckenmester, W., Prahm, C., Mutze, S., Ekkernkamp, A., & Hönning, A. (2026) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd016266 · Cited by 1
Li, J., Zhao, Y., Yang, Y., Ruan, W., Dang, R., Zhao, W., An, H., Xu, L., Tan, H., Guo, X., et al. (2026) · Infection · doi:10.1007/s15010-026-02741-x · Cited by 1
Metadata
- Authors
- Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., Weir-McCall, J. R., Teng, Z., Gkrania-Klotsas, E., AIX-COVNET, Ruggiero, A., Korhonen, A., Jefferson, E., Ako, E., Langs, G., Gozaliasl, G., Yang, G., Prosch, H., Preller, J., Stanczuk, J., Tang, J., Hofmanninger, J., Babar, J., Sánchez, L. E., Thillai, M., Gonzalez, P. M., Teare, P., Zhu, X., Patel, M., Cafolla, C., Azadbakht, H., Jacob, J., Lowe, J., Zhang, K., Bradley, K., Wassin, M., Holzer, M., Ji, K., Ortet, M. D., Ai, T., Walton, N., Lio, P., Stranks, S., Shadbahr, T., Lin, W., Zha, Y., Niu, Z., Rudd, J. H. F., Sala, E., & Schönlieb, C.-B.
- Affiliations
- AstraZeneca (United Kingdom) (GB)
- University of Cambridge (GB)
- University of Manchester (GB)
- Royal Papworth Hospital (GB)
- University of Dundee (GB)
- Royal Brompton Hospital (GB)
- Medical University of Vienna (AT)
- University of Helsinki (FI)
- Imperial College London (GB)
- Boehringer Ingelheim (Germany) (DE)
- Addenbrooke's Hospital (GB)
- University College London (GB)
- Macau University of Science and Technology (MO)
- Tongji Hospital (CN)
- Renmin Hospital of Wuhan University (CN)
- Corresponding author
- Roberts, Michael
- ORCID
- Roberts, Michael
- Thorpe, Matthew
- Gilbey, Julian
- Yeung, Michael
- Ursprung, Stephan
- McCague, Cathal
- Weir-McCall, Jonathan R.
- Gkrania-Klotsas, Effrossyni
- Ako, Emmanuel
- Langs, Georg
- Gozaliasl, Ghassem
- Yang, Guang
- Prosch, Helmut
- Preller, Jacobus
- Azadbakht, Hojjat
- Jacob, Joseph
- Ji, Kangyu
- Walton, Nicholas
- Stranks, Samuel
- Shadbahr, Tolou
- Rudd, James H. F.
- Sala, Evis
- Schönlieb, Carola-Bibiane
- Venue
- Nature Machine Intelligence
- Published
- 2021-03-15
- Volume
- 3
- Issue
- 3
- Pages
- 199-217
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W3119527628
- Cited by
- 1,044
- References
- 83
- Citations by year
- 2026: 203 · 2025: 96 · 2024: 143 · 2023: 248 · 2022: 224 · 2021: 130
- FWCI
- 98.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Machine Intelligence (publishedVersion, cc-by)
- Discovery Research Portal (University of Dundee) (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- elib (German Aerospace Center) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Työväentutkimus Vuosikirja (submittedVersion)
- arXiv (Cornell University)
- Funders
- AstraZeneca
- European Commission (777826)
- Medical Research Council (1966157, MR/S010351/1)
- Engineering and Physical Sciences Research Council (EP/T017961/1, EP/N014588/1, EP/S026045/1)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- COVID-19 pandemic, machine learning, chest radiograph, chest CT, COVID-19 detection, outcome prediction, clinical utility, model validation, methodological errors, bias, research recommendations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
