The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.
Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
Open accessAbstract
Abstract Machine learning methods offer great promise for fast and accurate detection and prognostication of coronavirus disease 2019 (COVID-19) from standard-of-care chest radiographs (CXR) and chest computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we consider all published papers and preprints, for the period from 1 January 2020 to 3 October 2020, which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. All manuscripts uploaded to bioRxiv, medRxiv and arXiv along with all entries in EMBASE and MEDLINE in this timeframe are considered. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher-quality model development and well-documented manuscripts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Much of the current evidence is based on retrospective analyses with limited prospective validation [19,28].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The success of deep learning in computer vision (CV) analysis has significantly advanced its clinical applications in comprehending biomedical images, including CT scans [88], positron emission tomography (PET)-CT scans [89], whole-slide images (WSIs) [90], and MRI scans [91].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [88]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Differences in image acquisition and processing may result in “shortcuts” where models take cues from non-clinical features.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“Based on this study, we suggest that such a gold rush may give rise to un-engaged 'augmentation' which is highly risky from a learning and knowledge perspective for experts, AI companies, and consumers.”
From To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis (2022) · cites this paper as Roberts et al. 2021
In the citing paper
“Despite problems like the one discovered by Roberts et al. [195], the fact that ML systems process and analyze data differently from human beings can be a great benefit to science and society because these systems may be able to identify useful and innovative structures, properties, patterns, and relationships that human beings would not recognize.”
From The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool (2025) · cites this paper as [195]
In the citing paper
“For example, COVID-19 prediction models have been demonstrated to learn which institution the images were obtained from rather than features pertaining to the underlying disease. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“The methodological flaws likely is a reflection of the lack of diversity in the development team to allow clinicians, data analysts and patient advocacy groups to work side-by-side throughout the AI life cycle. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“A similar result is summarized in a systematic review of COVID-19 prediction models on CT scans and Chest X-rays (CXRs) that found limited clinical utility due to methodological flaws and underlying biases stemming from small training data sets, data set variability, and limited integration of non-imaging data, among others.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Wang, D., Hu, B., Hu, C., Zhu, F., Liu, X., Zhang, J., Wang, B., Xiang, H., Cheng, Z., Xiong, Y., et al. (2020) · JAMA · doi:10.1001/jama.2020.1585 · Cited by 20,148
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Ai, T., Yang, Z., Hou, H., Zhan, C., Chen, C., Lv, W., Tao, Q., Sun, Z., & Xia, L. (2020) · Radiology · doi:10.1148/radiol.2020200642 · Cited by 5,793
Self-citationRadiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688
Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning
Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C. S., Liang, H., Baxter, S. L., McKeown, A., Yang, G., Wu, X., Yan, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.010 · Cited by 4,842
Self-citation
Research works printed but not linked: 7
- Wong, H. Y. F. et al. Frequency and distribution of chest radiographic findings in COVID-19 positive patients. Radiology 296, 201160 (2019).
- Bassi, P. R. A. S. & Attux, R. A deep convolutional neural network for COVID-19 detection using chest X-rays. Preprint at http://arxiv.org/abs/2005.01578 (2020). · arXiv
- Amyar, A., Modzelewski, R. & Ruan, S. Multi-task deep learning based CT imaging analysis for COVID-19: classification and segmentation. Preprint at medRxiv https://doi.org/10.1101/2020.04.16.2006470 (2020). · DOI
- Goodfellow, I. J. et al. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems Vol. 2 2672–2680 (MIT Press, 2014).
- Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations ICLR 2017 (ICLR, 2017).
- Covidence Systematic Review Software v2486 6284093b (Veritas Health Innovation, 2020); https://www.covidence.org/
- Aviles-Rivero, A. I., Sellars, P., Schönlieb, C.-B. & Papadakis, N. GraphXCOVID: explainable deep graph diffusion pseudo-labelling for identifying COVID-19 on chest X-rays. Preprint at https://arxiv.org/abs/2010.00378 (2020). · arXiv
Cited by
Sorted by citations
Items 101–200 of 1,000
Translating AI to Clinical Practice: Overcoming Data Shift with Explainability
Choi, Y., Yu, W., Nagarajan, M. B., Teng, P., Goldin, J. G., Raman, S. S., Enzmann, D. R., Kim, G. H. J., & Brown, M. S. (2023) · RadioGraphics · doi:10.1148/rg.220105 · Cited by 65
Akudjedu, T. N., Torre, S., Khine, R., Katsifarakis, D., Newman, D., & Malamateniou, C. (2022) · Journal of medical imaging and radiation sciences · doi:10.1016/j.jmir.2022.11.016 · Cited by 68
Boundary Guided Semantic Learning for Real-Time COVID-19 Lung Infection Segmentation System
Cong, R., Zhang, Y., Yang, N., Li, H., Zhang, X., Li, R., Chen, Z., Zhao, Y., & Kwong, S. (2022) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2022.3205376 · Cited by 64
“design, as described in recent and important review papers [19], [20], [21], some common-sense pitfalls and biases are waiting to be solved, including the data used”
BackgroundMethodThe use of machine learning and artificial intelligence within pediatric critical care
Shah, N. A., Arshad, A., Mazer, M. B., Carroll, C. L., Shein, S. L., & Remy, K. E. (2022) · Pediatric Research · doi:10.1038/s41390-022-02380-6 · Cited by 70
“Specific examples range from sepsis prediction to imaging classification.(79,97,98) During development particular focus needs to be paid to the definitions and scope of the model, and the selection of predictor variables.”
BackgroundReview on the Evaluation and Development of Artificial Intelligence for COVID-19 Containment
Hasan, M. M., Islam, M. U., Sadeq, M. J., Fung, W.-K., & Uddin, J. (2023) · Sensors · doi:10.3390/s23010527 · Cited by 70
“[20] highlighted medical image processing applications for COVID-19 that use deep learning and machine learning methods.”
BackgroundCOVID Mortality Prediction with Machine Learning Methods: A Systematic Review and Critical Appraisal
Bottino, F., Tagliente, E., Pasquini, L., Napoli, A. D., Lucignani, M., Figà-Talamanca, L., & Napolitano, A. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11090893 · Cited by 61
“Recent studies reviewed predictive models for SARS-CoV-2 diagnosis and severity, length of hospital stay, intensive care unit (ICU) admission, mechanical ventilation modality outcomes [8–12], highlighting pitfalls of the machine and deep learning methods based on imaging data [13]; however, systematic reviews focused…”
BackgroundMethodInfluentialApplying Different Machine Learning Techniques for Prediction of COVID-19 Severity
Sayed, S. A.-F., ElKorany, A., & Sayed, S. (2021) · IEEE Access · doi:10.1109/access.2021.3116067 · Cited by 63
“This paper follows most of the proposed recommendations in [12], like implementation replicability, where the source code has been uploaded on GitHub at https://github.com/safynaz/Cov-Sev,.”
BackgroundMethodA Self-Activated CNN Approach for Multi-Class Chest-Related COVID-19 Detection
Rehman, N.-U., Zia, M. S., Meraj, T., Rauf, H. T., Damaševičius, R., El‐Sherbeeny, A. M., & El-Meligy, M. A. (2021) · Applied Sciences · doi:10.3390/app11199023 · Cited by 59
“Artificial intelligence (AI) techniques guarantee quick and exact identification and forecasts of COVID-19 from standard-of-care chest radiographs and other processed tomography chest images [18].”
BackgroundAssessment of Radiology Artificial Intelligence Software: A Validation and Evaluation Framework
Tanguay, W., Acar, P., Fine, B., Abdolell, M., Gong, B., Cadrin-Chênevert, A., Chartrand‐Lefebvre, C., Chalaoui, J., Gorgos, A. B., Chin, A. S.-L., et al. (2022) · Canadian Association of Radiologists Journal · doi:10.1177/08465371221135760 · Cited by 62
“Yet, in a large systematic review of COVID-19 detection algorithms, 258 of 320 publications screened using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) and Radiomics Quality Score (RQS) failed quality control, with insufficient documentation of model selection, image pre-processing and algorith…”
BackgroundSS-TBN: A Semi-Supervised Tri-Branch Network for COVID-19 Screening and Lesion Segmentation
Zeng, L., Gao, K., Hu, D., Feng, Z., Hou, C., Rong, P., & Wang, W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3240886 · Cited by 58
“Machine learning methods, especially deep neural networks (DNNs), offer great promise for fast and accurate detection of COVID-19 infections from CT images [6], [7], despite potential methodological flaws and underlying biases [8].”
Garcia Santa Cruz, B., Bossa, M. N., Sölter, J., & Husch, A. D. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102225 · Cited by 89
Advancing AI Data Ethics in Nursing: Future Directions for Nursing Practice, Research, and Education
Dunlap, P. A. B., & Michalowski, M. (2024) · JMIR Nursing · doi:10.2196/62678 · Cited by 70
“Moreover, using suboptimal data to train machine learning algorithms can result in harmful outcomes for patients, such as misdiagnosis [18], misidentification [18,19], or data privacy breaches, even if the data were deidentified before use in machine learning algorithms [8].”
Good scientific practice in EEG and MEG research: Progress and perspectives
Niso, G., Krol, L. R., Combrisson, E., Dubarry, A., Elliott, M., Clément, F., Héjja‐Brichard, Y., Herbst, S. K., Jerbi, K., Ković, V., et al. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119056 · Cited by 68
COVID-CT-Mask-Net: prediction of COVID-19 from CT scans using regional features
Ter-Sarkisov, A. (2022) · Applied Intelligence · doi:10.1007/s10489-021-02731-6 · Cited by 56
“methodological flaws discussed in [24], and our solution can be both verified and used in other studies.”
BackgroundResultInfluentialSharma, A., & Mishra, P. K. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108826 · Cited by 56
Karpov, O. E., Николаевна, П. Е., Kurkin, S., Maksimenko, V., Gusev, A. V., Shusharina, N. N., & Hramov, A. E. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20075335 · Cited by 64
“The COVID-19 pandemic has significantly increased interest in the use of AI products, although, as noted in two recent meta-reviews [15,16], the effectiveness of AI-based models for predicting the severity of the COVID-19 disease has not been sufficiently high.”
BackgroundJones, M. A., Islam, W., Faiz, R., Chen, X., & Zheng, B. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.980793 · Cited by 61
“However, despite the challenge of how to achieve high scientific rigor when developing AI-based deep learning models (31), applying AI technology to develop CAD schemes has become the mainstream technique of the CAD research community.”
BackgroundMethodExpectations for Artificial Intelligence (AI) in Psychiatry
Monteith, S., Glenn, T., Geddes, J. R., Whybrow, P. C., Achtyes, E. D., & Bauer, M. (2022) · Current Psychiatry Reports · doi:10.1007/s11920-022-01378-5 · Cited by 61
CASPI: collaborative photon processing for active single-photon imaging
Lee, J., Ingle, A. N., Chacko, J. V., Eliceiri, K. W., & Gupta, M. (2023) · Nature Communications · doi:10.1038/s41467-023-38893-9 · Cited by 55
“Modern deep-learning-based approaches often do not generalize well for out-of-distribution settings, making practical deployment for mission-critical applications such as biomedical imaging challenging 34 .”
Soltan, A. A. S., Yang, J., Pattanshetty, R., Novak, A., Yang, Y., Rohanian, O., Beer, S., Soltan, M. A., Thickett, D. R., Fairhead, R., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00272-7 · Cited by 54
“25,26 A 2021 editorial highlighted the promise of CURIAL-1.”
Scantlebury, J., Vost, L., Carbery, A., Hadfield, T. E., Turnbull, O. M., Brown, N., Chenthamarakshan, V., Das, P., Grosjean, H., von Delft, F., et al. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c00322 · Cited by 54
Granlund, T., Stirbu, V., & Mikkonen, T. (2021) · SN Computer Science · doi:10.1007/s42979-021-00726-1 · Cited by 53
Moving towards vertically integrated artificial intelligence development
Zhang, J., Budhdeo, S., William, W., Cerrato, P., Shuaib, H., Sood, H., Ashrafian, H., Halamka, J., & Teo, J. T. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00690-x · Cited by 54
“However, a common feature of the clinical AI research translational gap remains separation of dataset experimentation with ability to operationalise models, as seen in lack of candidates for clinical translation amongst hundreds of COVID-19 models with high reported accuracy 60,61 .”
BackgroundCorrecting data imbalance for semi-supervised COVID-19 detection using X-ray chest images
Calderon-Ramirez, S., Shengxiang-Yang, Moemeni, A., Elizondo, D. A., Colreavy‐Donnelly, S., Chavarría-Estrada, L. F., & Molina‐Cabello, M. A. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107692 · Cited by 56
Sushentsev, N., Silva, N. M. D., Yeung, M., Barrett, T., Sala, E., Roberts, M. T., & Rundo, L. (2022) · Insights into Imaging · doi:10.1186/s13244-022-01199-3 · Cited by 53
“This approach, which we previously adopted for another review [57], allowed us to only include studies that are reproducible.”
MethodSelf-citationCOVID-19 Detection in CT/X-ray Imagery Using Vision Transformers
Rahhal, M. M. A., Bazi, Y., Jomaa, R. M., AlShibli, A., Alajlan, N., Mekhalfi, M. L., & Melgani, F. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm12020310 · Cited by 54
“A comprehensive review for machine learning techniques used for COVID-19 detection and classification based on CXR or CT images was provided in [38].”
MethodMAIC–10 brief quality checklist for publications using artificial intelligence and medical images
Cerdá-Alberich, L., Solana, J., Mallol, P., Ribas, G., García-Junco, M., Alberich‐Bayarri, Á., & Martí‐Bonmatí, L. (2023) · Insights into Imaging · doi:10.1186/s13244-022-01355-9 · Cited by 56
Acar, E., Şahin, E., & Yılmaz, İ. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06344-5 · Cited by 52
A comparison of machine learning algorithms in predicting COVID-19 prognostics
Üstebay, S., Sarmış, A., Kaya, G. K., & Sujan, M. A. (2022) · Internal and Emergency Medicine · doi:10.1007/s11739-022-03101-x · Cited by 51
Kumar, V., Singh, D., Kaur, M., & Damaševičius, R. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.564 · Cited by 50
Machine learning and pre-medical education
Kolachalama, V. B. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102313 · Cited by 55
“On the other hand, studies have also revealed common pitfalls related to the development of medical AI models [5], and potential issues with mandating model explainability as a requirement for clinical deployment [6].”
BackgroundInnovations in thoracic imaging: CT , radiomics, AI and x‐ray velocimetry
Vliegenthart, R., Fouras, A., Jacobs, C., & Papanikolaou, N. K. (2022) · Respirology · doi:10.1111/resp.14344 · Cited by 50
“55,56 The application options for AI in thoracic imaging are diverse and extensive.”
BackgroundArtificial intelligence for telemedicine diabetic retinopathy screening: a review
Nakayama, L. F., Ribeiro, L. Z., Novaes, F. D. C., Miyawaki, I. A., Miyawaki, A. E., de Oliveira, J. A. E., de Oliveira, T. V. F., Malerbi, F. K., Regatieri, C. V., Celi, L. A., et al. (2023) · Annals of Medicine · doi:10.1080/07853890.2023.2258149 · Cited by 49
“Biases are a growing concern in the development and deployment of Ai systems, with numerous reports of problematic deployments in sepsis diagnosis, cOviD and patient triage [27,36,37].”
BackgroundMachine Learning for Health: Algorithm Auditing & Quality Control
Oala, L., Murchison, A. G., Balachandran, P., Choudhary, S., Fehr, J., Leite, A. W., Goldschmidt, P. G., Johner, C., Schörverth, E. D. M., Nakasi, R., et al. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01783-y · Cited by 48
“Developers proposing new ML4H technologies sometimes promise to match or even surpass the performance of existing methods [58] yet the reality is often more complicated.”
BackgroundCao, L., & Liu, Q. (2024) · ACM Computing Surveys · doi:10.1145/3686150 · Cited by 52
SD-UNet: A Novel Segmentation Framework for CT Images of Lung Infections
Yin, S., Deng, H., Xu, Z., Zhu, Q., & Cheng, J. (2022) · Electronics · doi:10.3390/electronics11010130 · Cited by 48
Multiple stakeholders drive diverse interpretability requirements for machine learning in healthcare
Imrie, F., Davis, R. A., & van der Schaar, M. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00698-2 · Cited by 54
“In particular, shortcut learning 45 , where the model learns a spurious relationship, must be avoided, as seen in several ML applications in healthcare 4,46,47 .”
BackgroundLu, S.-C., Xu, C., Nguyen, C., Geng, Y., Pfob, A., & Sidey-Gibbons, C. J. (2022) · JMIR Medical Informatics · doi:10.2196/33182 · Cited by 53
“Recent commentary has drawn attention to various issues in the transparency, performance, and reproducibility of ML tools [25-27].”
BackgroundCVD-HNet: Classifying Pneumonia and COVID-19 in Chest X-ray Images Using Deep Network
Suganyadevi, S., & Seethalakshmi, V. (2022) · Wireless Personal Communications · doi:10.1007/s11277-022-09864-y · Cited by 57
“4 [53].”
Thiagarajan, J. J., Thopalli, K., Rajan, D., & Turaga, P. K. (2022) · Scientific Reports · doi:10.1038/s41598-021-04529-5 · Cited by 50
“Such models typically perform well on the observed data, including passing standard cross-validation tests, yet fail when deployed in the real-world 5,6 .”
MethodThe reproducibility issues that haunt health-care AI
Sohn, E. (2023) · Nature · doi:10.1038/d41586-023-00023-2 · Cited by 49
Imrie, F., Cebere, B., McKinney, E. F., & van der Schaar, M. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000276 · Cited by 53
Deep learning model for tongue cancer diagnosis using endoscopic images
Heo, J., Lim, J. H., Lee, H., Jang, J. Y., Shin, Y. S., Kim, D., Lim, J. Y., Park, Y. M., Koh, Y. W., Ahn, S., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10287-9 · Cited by 46
“Driks emphasized the problem of “Frankenstein datasets” 22 .”
BackgroundMachine learning modeling practices to support the principles of AI and ethics in nutrition research
Thomas, D. M., Kleinberg, S., Brown, A. W., Crow, M., Bastian, N. D., Reisweber, N., Lasater, R., Kendall, T., Shafto, P., Blaine, R., et al. (2022) · Nutrition & Diabetes · doi:10.1038/s41387-022-00226-y · Cited by 48
“In addition, a recent review of 62 studies that used machine learning to detect and predict COVID-19 from chest radiographs and CT scans found that every single study had a methodological flaw [11].”
BackgroundResultda Silva, H. E. C., Santos, G. N. M., Leite, A. F., Mesquita, C. R. M., Figueiredo, P. T. D. S., Stefani, C. M., & de Melo, N. S. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292063 · Cited by 61
What are the current challenges for machine learning in drug discovery and repurposing?
Aittokallio, T. A. (2022) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2022.2050694 · Cited by 48
“Strikingly, a recent report showed that none of the ML models developed using chest radiographs and CT scans so far are of potential clinical use for COVID-19 management, mainly due to methodological flaws and high or unclear risk of bias in the imaging datasets used for model training [3].”
ResultChen, Z., Liu, X., Yang, Q., Wang, Y.-J., Miao, K., Gong, Z., Yu, Y., Leonov, A., Liu, C., Feng, Z., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.1671 · Cited by 43
Optimizing human-centered AI for healthcare in the Global South
Okolo, C. T. (2022) · Patterns · doi:10.1016/j.patter.2021.100421 · Cited by 49
“27 Given the time frame and lack of validation experiments for many of these studies, it is unlikely that they would have been deployed in real-world systems.”
BackgroundPrime Time for Artificial Intelligence in Interventional Radiology
Seah, J. C. Y., Boeken, T., Sapoval, M. R., & Goh, G. S. (2022) · CardioVascular and Interventional Radiology · doi:10.1007/s00270-021-03044-4 · Cited by 46
“The training and testing datasets must also have defined inclusion and exclusion criteria to prevent ‘‘Frankenstein’’ datasets[48], where positive and negative cases are drawn from different sources, potentially leading to data leakage as the AI model may recognize features specific to the dataset…”
BackgroundAgarwal, S., Wood, D. A., Grzeda, M. T., Suresh, C., Din, M., Cole, J. H., Marc Modat, A., & Booth, T. C. (2023) · Clinical Neuroradiology · doi:10.1007/s00062-023-01291-1 · Cited by 52
“…review analysing the enormous number of recent studies where AI was used for the detection of COVID-19 using chest imaging, found that all 62 included studies had no potential clinical use due to methodological biases such as the use of unrepresentative datasets and insufficient validation [9].”
Current state and prospects of artificial intelligence in allergy
van Breugel, M., Fehrmann, R. S. N., Bügel, M., Rezwan, F. I., Holloway, J. W., Nawijn, M. C., Fontanella, S., Čustović, A., & Koppelman, G. H. (2023) · Allergy · doi:10.1111/all.15849 · Cited by 46
Classification of Alzheimer’s disease stages from magnetic resonance images using deep learning
Mora-Rubio, A., Bravo-Ortíz, M. A., Arredondo, S., Saborit-Torres, J. M., Ruz, G. A., & Tabares-Soto, R. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1490 · Cited by 43
“As reported in medical tasks, a signi fi cant portion of predictive models presented in the scienti fi c literature offer little to no value to the healthcare community due to the presence of bias in the models, which affects the accuracy of predictions ( Roberts et al., 2021 , Wynants et al., 2020 ).”
MethodDomain Shifts in Machine Learning Based Covid-19 Diagnosis From Blood Tests
Roland, T., Böck, C., Tschoellitsch, T., Maletzky, A., Hochreiter, S., Meier, J., & Klambauer, G. (2022) · Journal of Medical Systems · doi:10.1007/s10916-022-01807-1 · Cited by 42
Developing medical imaging AI for emerging infectious diseases
Huang, S.-C., Chaudhari, A. S., Langlotz, C. P., Shah, N. H., Yeung, S. Y., & Lungren, M. P. (2022) · Nature Communications · doi:10.1038/s41467-022-34234-4 · Cited by 42
Pinton, P. (2023) · Annals of Medicine · doi:10.1080/07853890.2023.2300670 · Cited by 42
“…diagnostics and clinical management, studies may present with methodological challenges such as small datasets and lack of external validation or comparison with healthcare professionals’ performance using the same sample, as well as limitations with study design, delivery, and reporting [70, 71].”
BackgroundA deep-learning-based framework for severity assessment of COVID-19 with CT images
Li, Z., Zhao, S., Chen, Y., Luo, F., Kang, Z., Cai, S., Zhao, W., Liu, J., Zhao, D., & Li, Y. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115616 · Cited by 42
“A large number of papers have been published since the pandemic outbroke (Roberts et al., 2021), and these studies can fall into three aspects: screening of COVID-19, infection segmentation, and severity assessment (Shi et al., 2021).”
BackgroundApplications of Neural Networks in Biomedical Data Analysis
Weiss, R., Karimijafarbigloo, S., Roggenbuck, D., & Rödiger, S. (2022) · Biomedicines · doi:10.3390/biomedicines10071469 · Cited by 51
“[79], where they analysed prediction models that had been recently described to support COVID-19 diagnosis (Wynant et al.”
MethodStatistical Learning Methods for Neuroimaging Data Analysis with Applications
Hongtu Zhu, B., Li, T., & Zhao, B. (2023) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-020722-100353 · Cited by 46
“Although the issue of sampling bias is prevalent in neuroimaging research, it has been largely ignored in the medical imaging literature until recently (57, 58).”
BackgroundMachine Learning in the Detection of Oral Lesions With Clinical Intraoral Images
Yasothkumar, D., Ramalingam, K., Ramani, P., & Deepak, R. M. (2023) · Cureus · doi:10.7759/cureus.44018 · Cited by 41
“Likewise, because of the heterogeneity of the clinical presentation of OPMDs and OSCC several sources and a multicentric approach with collaboration with several other cancer institutes must be considered [14].”
BackgroundMohan, G., Subashini, M. M., Balan, S., & Singh, S. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00110-x · Cited by 44
Discovery of a Generalization Gap of Convolutional Neural Networks on COVID-19 X-Rays Classification
Ahmed, K. B., Goldgof, G. M., Paul, R., Goldgof, D. B., & Hall, L. (2021) · IEEE Access · doi:10.1109/access.2021.3079716 · Cited by 43
“Authors in [46] questioned the clinical utility of the reviewed papers and discussed their methodological flaws.”
BackgroundInfluentialGenomics enters the deep learning era
Routhier, E., & Mozziconacci, J. (2022) · PeerJ · doi:10.7717/peerj.13613 · Cited by 45
Developing robust benchmarks for driving forward AI innovation in healthcare
Mincu, D., & Roy, S. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00559-4 · Cited by 41
“However, such models are likely to overperform as they are merely detecting children versus adults, and are inherently biased 19 .”
BackgroundBalaha, H. M., El-Gendy, E. M., & Saafan, M. M. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10127-8 · Cited by 40
“CT image dataset In their comprehensive review, Roberts (2021) recommended using not only data from the Internet but also adding new high-quality data to avoid overfitting and to solve the bias problem.”
MethodData-Centric and Model-Centric AI: Twin Drivers of Compact and Robust Industry 4.0 Solutions
Hamid, O. H. (2023) · Applied Sciences · doi:10.3390/app13052753 · Cited by 46
Spectral imaging enables contrast agent–free real-time ischemia monitoring in laparoscopic surgery
Ayala, L. A., Adler, T. J., Seidlitz, S., Wirkert, S., Engels, C., Seitel, A., Sellner, J., Aksenov, A., Bodenbach, M., Bader, P., et al. (2023) · Science Advances · doi:10.1126/sciadv.add6778 · Cited by 40
Machine Learning in Tremor Analysis: Critique and Directions
De, A., Bhatia, K. P., Volkmann, J., Peach, R. L., & Schreglmann, S. R. (2023) · Movement Disorders · doi:10.1002/mds.29376 · Cited by 44
Chowdhury, A. H., Rad, D., & Rahman, M. S. (2024) · Health Science Reports · doi:10.1002/hsr2.2037 · Cited by 55
“39,40 Healthcare is one area where ML has enormous promise.”
BackgroundPerformance of a Chest Radiograph AI Diagnostic Tool for COVID-19: A Prospective Observational Study
Sun, J., Peng, L., Li, T., Adila, D., Zaiman, Z., Melton, G. B., Ingraham, N. E., Murray, E., Boley, D. L., Switzer, S., et al. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.210217 · Cited by 39
“A recent review of 62 AI models for COVID-19 from biomedical imaging identified substantial limitations in AI models published to date and identified a majority of articles as being at high risk of bias (5).”
BackgroundAutomated Tumor Segmentation in Radiotherapy
Savjani, R. R., Lauria, M., Bose, S., Deng, J., Yuan, Y., & Andrearczyk, V. (2022) · Seminars in Radiation Oncology · doi:10.1016/j.semradonc.2022.06.002 · Cited by 44
A historical perspective of biomedical explainable AI research
Malinverno, L., Barros, V., Ghisoni, F., Visonà, G., Kern, R., Nickel, P. J., Ventura, B. E., Šimić, I., Stryeck, S., Manni, F., et al. (2023) · Patterns · doi:10.1016/j.patter.2023.100830 · Cited by 46
“12,13 It is worth noting that across the most cited papers we reviewed, often the leading author (last co-author) was an AI researcher that expanded their horizon to the applied biomedical field.”
BackgroundWilliams, M. C., Weir-McCall, J. R., Baldassarre, L. A., Cecco, C. N. D., Choi, A. D., Dey, D., Dweck, M. R., Išgum, I., Kolossváry, M., Leipsic, J. A., et al. (2024) · Journal of cardiovascular computed tomography · doi:10.1016/j.jcct.2024.08.003 · Cited by 38
Self-citationA multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis
Liu, F., Zheng, L., Yin, Q., Huang, J., Luo, J., Thakur, A., Branson, K., Schwab, P., Yin, B., Wu, X., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01339-7 · Cited by 50
Evaluating transparency in AI/ML model characteristics for FDA-reviewed medical devices
Mehta, V., Komanduri, A., Bhadouriya, R. S., Mehta, V., Johnson, M. D., Shrestha, P., Nikolov, M. C., Jain, B., Shah, N. H., & Schulman, K. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02052-9 · Cited by 47
Peng, L., Wang, C., Tian, G., Liu, G., Gan, L., Lu, Y., Yang, J., Chen, M., & Li, Z. (2022) · Frontiers in Microbiology · doi:10.3389/fmicb.2022.995323 · Cited by 38
MLATE: Machine learning for predicting cell behavior on cardiac tissue engineering scaffolds
Rafieyan, S., Vasheghani‐Farahani, E., Baheiraei, N., & Keshavarz, H. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106804 · Cited by 43
Grand challenges in bioinformatics education and training
Işık, E. B., Brazas, M. D., Schwartz, R. S., Gaëta, B. A., Palagi, P. M., van Gelder, C. W. G., Suravajhala, P., Singh, H., Morgan, S., Zahroh, H., et al. (2023) · Nature Biotechnology · doi:10.1038/s41587-023-01891-9 · Cited by 38
Avilés-Rivero, A. I., Sellars, P., Schönlieb, C., & Papadakis, N. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108274 · Cited by 39
Self-citationRobust weakly supervised learning for COVID-19 recognition using multi-center CT images
Ye, Q., Gao, Y., Ding, W., Niu, Z., Wang, C., Jiang, Y., Wang, M., Fang, E. F., Menpes-Smith, W., Xia, J., et al. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.108291 · Cited by 38
Self-citationKim, C. K., Choi, J. W., Jiao, Z., Wang, D., Wu, J., Yi, T. Y., Halsey, K., Eweje, F. R., Tran, T. M. L., Liu, C., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00546-w · Cited by 36
“Earlier reviews have published predictive models for COVID-19 diagnosis and prognosis, but many of these studies present numerous sources of potential bias that the proposed study addresses 20 (Supplementary Discussion).”
InfluentialMeng, Y., Bridge, J., Addison, C. A., Wang, M., Merritt, C., Franks, S., Mackey, M., Messenger, S., Sun, R., FitzMaurice, T. S., et al. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102722 · Cited by 35
“Despite the large number of deep learning models that have been proposed so far for the diagnosis of COVID-19 using computed tomography (CT) and X-ray , none of them is clinically useable due to methodological flaws and/or underlying biases (Roberts et al., 2021).”
BackgroundFedorov, A., Longabaugh, W. J. R., Pot, D., Clunie, D. A., Pieper, S. D., Gibbs, D. L., Bridge, C., Herrmann, M. D., Homeyer, A., Lewis, R., et al. (2023) · Radiographics · doi:10.1148/rg.230180 · Cited by 35
“Insufficient or unknown provenance of the datasets used while developing ML imaging tools plagues many, if not most, research studies (35), in turn jeopardizing transparency of those studies and reducing their reproducibility.”
Huo, W., Zhang, Z., Qu, J., Yan, J., Yan, S., Yan, J., & Shi, B. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2176985 · Cited by 41
“Furthermore, some algorithmic models do not have many high clinical application implications, which can stem from unstable results and the misclassification of particularly simple examples (Finlayson et al., 2019; Roberts et al., 2021).”
BackgroundAlle, S., Kanakan, A., Siddiqui, S., Garg, A., Karthikeyan, A., Mehta, P., Mishra, N., Chattopadhyay, P., Devi, P., Waghdhare, S., et al. (2022) · PLoS ONE · doi:10.1371/journal.pone.0264785 · Cited by 35
“COVID-19 diagnosis from radiographs and scans, none were found to be reproducible and transparent enough to be deployed in clinical settings [51].”
BackgroundResultSelf-citationDoyen, S., & Dadario, N. B. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.765406 · Cited by 37
“Indeed, thousands of exciting models with increasing collaboration and data sharing had been immediately proposed throughout the pandemic to improve the prognostication and clinical management of COVID-19 patients (7, 8).”
BackgroundMethodInfluentialContinental generalization of a human-in-the-loop AI system for clinical seizure recognition
Yang, Y., Truong, N. D., Maher, C., Nikpour, A., & Kavehei, O. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118083 · Cited by 39
The unintended consequences of artificial intelligence in paediatric radiology
Ciet, P., Eade, C., Ho, M., Laborie, L. B., Mahomed, N., Naidoo, J., Pace, E., Segal, B., Toso, S., Tschauner, S., et al. (2023) · Pediatric Radiology · doi:10.1007/s00247-023-05746-y · Cited by 34
“…is liable for errors Lack of transparency “Black box” concept of how AI tools are coming to their decisions may mean lack of trust for the output or explanation provided review analysed over two thousand AI model publications and found that none of them were adequate to serve a clinical role [9].”
AI-aided on-chip nucleic acid assay for smart diagnosis of infectious disease
Sun, H. R., Xiong, L., Huang, Y., Chen, X., Yu, Y., Shao-zhen, Y., Dong, H., Yuan, J. Y., & Zhang, W. (2021) · Fundamental Research · doi:10.1016/j.fmre.2021.12.005 · Cited by 34
Ho, A. (2023) · doi:10.1093/med/9780197556269.001.0001 · Cited by 35
Rahman, T., Chowdhury, M. E. H., Khandakar, A. A., Mahbub, Z. B., Hossain, M. S. A., Alhatou, A., Abdalla, E. O. I., Muthiyal, S., Islam, K. F., Kashem, S. B. A., et al. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08606-w · Cited by 34
New trend in artificial intelligence-based assistive technology for thoracic imaging
Yanagawa, M., Ito, R., Nozaki, T., Fujioka, T., Yamada, A., Fujita, S., Kamagata, K., Fushimi, Y., Tsuboyama, T., Matsui, Y., et al. (2023) · La radiologia medica · doi:10.1007/s11547-023-01691-w · Cited by 33
“However, various AI models for COVID-19 pneumonia classification using chest radiography and chest CT that have appeared since the early days of the COVID-19 have been in a mixed state, with conspicuous variations in databases and analysis methods [80, 93].”
Babu, A., & Mandal, D. (2024) · ACS Applied Energy Materials · doi:10.1021/acsaem.3c02642 · Cited by 33
Kwong, J. C. C., Wu, J., Malik, S., Khondker, A., Gupta, N. K., Bodnariuc, N., Narayana, K., Malik, M., van der Kwast, T., Johnson, A. E. W., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01088-7 · Cited by 40
Vervoort, D., Tam, D. Y., & Wijeysundera, H. C. (2021) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2021.08.015 · Cited by 37
Haymond, S., & Master, S. R. (2021) · Clinical Chemistry · doi:10.1093/clinchem/hvab272 · Cited by 33
A Review of the Potential of Artificial Intelligence Approaches to Forecasting COVID-19 Spreading
Jamshidi, M. B., Roshani, S., Talla, J., Lalbakhsh, A., Peroutka, Z., Roshani, S., Parandin, F., Malek, Z., Daneshfar, F., Niazkar, H. R., et al. (2022) · AI · doi:10.3390/ai3020028 · Cited by 33
“Ironically, despite the considerable potential of AI applications in fighting COVID-19, the current literature suggests that relevant studies may not be clinically relevant [60].”
BackgroundGleeson, F. V., Revel, M., Biederer, J., Larici, A. R., Martini, K., Frauenfelder, T., Screaton, N. J., Prosch, H., Snoeckx, A., Sverzellati, N., et al. (2023) · European Radiology · doi:10.1007/s00330-023-09409-2 · Cited by 35
“A systematic review of 62 published studies on ML to detect and/or prognosticate COVID-19 revealed that none were without methodological flaws or biases, consequently none of the studies were deemed clinically useful [8].”
BackgroundSelf-citationProceedings of the NHLBI Workshop on Artificial Intelligence in Cardiovascular Imaging
Dey, D., Arnaout, R., Antani, S., Badano, A. G., Jacques, L., Li, H., Leiner, T., Margerrison, E., Samala, R. K., Sengupta, P. P., et al. (2023) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2023.05.012 · Cited by 36
Replicability and generalizability in population psychiatric neuroimaging
Marek, S. A., & Laumann, T. O. (2024) · Neuropsychopharmacology · doi:10.1038/s41386-024-01960-w · Cited by 34
“A concerning number of such events have transpired in the medical imaging community, including the use of machine learning to identify lungs with pneumonia [47] and Covid-19 [48].”
Raghu Raman, R., Vinuesa, R., & Nedungadi, P. (2021) · Sustainability · doi:10.3390/su13147555 · Cited by 31
Metadata
- Authors
- Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., Weir-McCall, J. R., Teng, Z., Gkrania-Klotsas, E., AIX-COVNET, Ruggiero, A., Korhonen, A., Jefferson, E., Ako, E., Langs, G., Gozaliasl, G., Yang, G., Prosch, H., Preller, J., Stanczuk, J., Tang, J., Hofmanninger, J., Babar, J., Sánchez, L. E., Thillai, M., Gonzalez, P. M., Teare, P., Zhu, X., Patel, M., Cafolla, C., Azadbakht, H., Jacob, J., Lowe, J., Zhang, K., Bradley, K., Wassin, M., Holzer, M., Ji, K., Ortet, M. D., Ai, T., Walton, N., Lio, P., Stranks, S., Shadbahr, T., Lin, W., Zha, Y., Niu, Z., Rudd, J. H. F., Sala, E., & Schönlieb, C.-B.
- Affiliations
- AstraZeneca (United Kingdom) (GB)
- University of Cambridge (GB)
- University of Manchester (GB)
- Royal Papworth Hospital (GB)
- University of Dundee (GB)
- Royal Brompton Hospital (GB)
- Medical University of Vienna (AT)
- University of Helsinki (FI)
- Imperial College London (GB)
- Boehringer Ingelheim (Germany) (DE)
- Addenbrooke's Hospital (GB)
- University College London (GB)
- Macau University of Science and Technology (MO)
- Tongji Hospital (CN)
- Renmin Hospital of Wuhan University (CN)
- Corresponding author
- Roberts, Michael
- ORCID
- Roberts, Michael
- Thorpe, Matthew
- Gilbey, Julian
- Yeung, Michael
- Ursprung, Stephan
- McCague, Cathal
- Weir-McCall, Jonathan R.
- Gkrania-Klotsas, Effrossyni
- Ako, Emmanuel
- Langs, Georg
- Gozaliasl, Ghassem
- Yang, Guang
- Prosch, Helmut
- Preller, Jacobus
- Azadbakht, Hojjat
- Jacob, Joseph
- Ji, Kangyu
- Walton, Nicholas
- Stranks, Samuel
- Shadbahr, Tolou
- Rudd, James H. F.
- Sala, Evis
- Schönlieb, Carola-Bibiane
- Venue
- Nature Machine Intelligence
- Published
- 2021-03-15
- Volume
- 3
- Issue
- 3
- Pages
- 199-217
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W3119527628
- Cited by
- 1,044
- References
- 83
- Citations by year
- 2026: 203 · 2025: 96 · 2024: 143 · 2023: 248 · 2022: 224 · 2021: 130
- FWCI
- 98.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Machine Intelligence (publishedVersion, cc-by)
- Discovery Research Portal (University of Dundee) (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- elib (German Aerospace Center) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Työväentutkimus Vuosikirja (submittedVersion)
- arXiv (Cornell University)
- Funders
- AstraZeneca
- European Commission (777826)
- Medical Research Council (1966157, MR/S010351/1)
- Engineering and Physical Sciences Research Council (EP/T017961/1, EP/N014588/1, EP/S026045/1)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- COVID-19 pandemic, machine learning, chest radiograph, chest CT, COVID-19 detection, outcome prediction, clinical utility, model validation, methodological errors, bias, research recommendations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
