The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.
Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
Open accessAbstract
Abstract Machine learning methods offer great promise for fast and accurate detection and prognostication of coronavirus disease 2019 (COVID-19) from standard-of-care chest radiographs (CXR) and chest computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we consider all published papers and preprints, for the period from 1 January 2020 to 3 October 2020, which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. All manuscripts uploaded to bioRxiv, medRxiv and arXiv along with all entries in EMBASE and MEDLINE in this timeframe are considered. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher-quality model development and well-documented manuscripts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Much of the current evidence is based on retrospective analyses with limited prospective validation [19,28].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The success of deep learning in computer vision (CV) analysis has significantly advanced its clinical applications in comprehending biomedical images, including CT scans [88], positron emission tomography (PET)-CT scans [89], whole-slide images (WSIs) [90], and MRI scans [91].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [88]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Differences in image acquisition and processing may result in “shortcuts” where models take cues from non-clinical features.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“Based on this study, we suggest that such a gold rush may give rise to un-engaged 'augmentation' which is highly risky from a learning and knowledge perspective for experts, AI companies, and consumers.”
From To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis (2022) · cites this paper as Roberts et al. 2021
In the citing paper
“Despite problems like the one discovered by Roberts et al. [195], the fact that ML systems process and analyze data differently from human beings can be a great benefit to science and society because these systems may be able to identify useful and innovative structures, properties, patterns, and relationships that human beings would not recognize.”
From The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool (2025) · cites this paper as [195]
In the citing paper
“For example, COVID-19 prediction models have been demonstrated to learn which institution the images were obtained from rather than features pertaining to the underlying disease. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“The methodological flaws likely is a reflection of the lack of diversity in the development team to allow clinicians, data analysts and patient advocacy groups to work side-by-side throughout the AI life cycle. 15”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
In the citing paper
“A similar result is summarized in a systematic review of COVID-19 prediction models on CT scans and Chest X-rays (CXRs) that found limited clinical utility due to methodological flaws and underlying biases stemming from small training data sets, data set variability, and limited integration of non-imaging data, among others.”
From AI pitfalls and what not to do: mitigating bias in AI (2023) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Wang, D., Hu, B., Hu, C., Zhu, F., Liu, X., Zhang, J., Wang, B., Xiang, H., Cheng, Z., Xiong, Y., et al. (2020) · JAMA · doi:10.1001/jama.2020.1585 · Cited by 20,148
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Ai, T., Yang, Z., Hou, H., Zhan, C., Chen, C., Lv, W., Tao, Q., Sun, Z., & Xia, L. (2020) · Radiology · doi:10.1148/radiol.2020200642 · Cited by 5,793
Self-citationRadiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688
Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning
Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C. S., Liang, H., Baxter, S. L., McKeown, A., Yang, G., Wu, X., Yan, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.010 · Cited by 4,842
Self-citation
Research works printed but not linked: 7
- Wong, H. Y. F. et al. Frequency and distribution of chest radiographic findings in COVID-19 positive patients. Radiology 296, 201160 (2019).
- Bassi, P. R. A. S. & Attux, R. A deep convolutional neural network for COVID-19 detection using chest X-rays. Preprint at http://arxiv.org/abs/2005.01578 (2020). · arXiv
- Amyar, A., Modzelewski, R. & Ruan, S. Multi-task deep learning based CT imaging analysis for COVID-19: classification and segmentation. Preprint at medRxiv https://doi.org/10.1101/2020.04.16.2006470 (2020). · DOI
- Goodfellow, I. J. et al. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems Vol. 2 2672–2680 (MIT Press, 2014).
- Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations ICLR 2017 (ICLR, 2017).
- Covidence Systematic Review Software v2486 6284093b (Veritas Health Innovation, 2020); https://www.covidence.org/
- Aviles-Rivero, A. I., Sellars, P., Schönlieb, C.-B. & Papadakis, N. GraphXCOVID: explainable deep graph diffusion pseudo-labelling for identifying COVID-19 on chest X-rays. Preprint at https://arxiv.org/abs/2010.00378 (2020). · arXiv
Cited by
Sorted by citations
Items 301–400 of 1,000
Tzeng, I., Hsieh, P., Su, W., Hsieh, T.-H. J., & Chang, S.-C. (2023) · Diagnostics · doi:10.3390/diagnostics13040584 · Cited by 20
“Thus, evaluating the performance of externally checked models was superior to disregarding externally checked models [42].”
ResultBiases associated with database structure for COVID-19 detection in X-ray images
Arias-Garzón, D., Tabares-Soto, R., Bernal-Salcedo, J., & Ruz, G. A. (2023) · Scientific Reports · doi:10.1038/s41598-023-30174-1 · Cited by 18
Real-Time Prediction for Neonatal Endotracheal Intubation Using Multimodal Transformer Network
Im, J.-E., Yoon, S.-A., Shin, Y. M., & Park, S. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3267521 · Cited by 16
“ing (XGBoost) [38], [39], [40], SVM [36], [41], [42], MLP [36], [43], [44], [45] and LSTM [46], [47], [48], [49].”
MethodMeasurement Methodology: Visualizing Uncertainty in Machine Learning-Assisted Measurements
Shirmohammadi, S. (2023) · IEEE Instrumentation & Measurement Magazine · doi:10.1109/mim.2023.10238361 · Cited by 16
“The result? “Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases” [2].”
BackgroundBaughan, N. M., Whitney, H. M., Drukker, K., Sahiner, B., Hu, T., Kim, G. H. J., McNitt‐Gray, M. F., Myers, K. J., & Giger, M. L. (2023) · Journal of medical imaging · doi:10.1117/1.jmi.10.6.064501 · Cited by 16
“The variables of age, race, sex at birth, ethnicity, COVID-19 status, and image modality contained 9, 7, 4, 3, 3, and 4 categories, as listed in Table 1, respectively, resulting in a total of 9072 strata.”
BackgroundAccelerating AI for science: open data science for science
Lawrence, N. D., & Montgomery, J. (2024) · Royal Society Open Science · doi:10.1098/rsos.231130 · Cited by 21
“…with the intention of improving healthcare outcomes, problems with data quality, methodological issues in the design of AI models and deficiencies in reporting practices all contributed to the development of a suite of AI systems that were generally unfit for use in clinical settings [14–16].”
BackgroundInterdisciplinary research in artificial intelligence: Lessons from COVID-19
Abbonato, D., Bianchini, S., Gargiulo, F., & Venturini, T. (2024) · Quantitative Science Studies · doi:10.1162/qss_a_00329 · Cited by 16
Enhancing Nanomaterial-Based Optical Spectroscopic Detection of Cancer through Machine Learning
Sahli, C., & Kenry, K. (2024) · ACS Materials Letters · doi:10.1021/acsmaterialslett.4c01267 · Cited by 15
RadEdit: Stress-Testing Biomedical Vision Models via Diffusion Image Editing
Pérez‐García, F., Bond-Taylor, S., Sanchez, P. P., van Breugel, B., Castro, D. C., Sharma, H., Salvatelli, V., Wetscherek, M. T., Richardson, H., Lungren, M. P., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73254-6_21 · Cited by 19
“Biomedical vision models, when put into real-world use, can be unhelpful or potentially even harmful to patients if they are affected by dataset shifts—leading to missed diagnoses [24, 60, 80, 82].”
BackgroundHoang, T.-H., Fuhrman, J. D., Klarqvist, M. D. R., Li, M., Chaturvedi, P., Li, Z., Kim, K., Ryu, M., Chard, R., Huerta, E. A., et al. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.12.001 · Cited by 20
I Find Your Lack of Uncertainty in Computer Vision Disturbing
Valdenegro-Toro, M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00139 · Cited by 15
“Image Classification COVID19 Detection AlexNet None from Chest X-Ray [1] VGG19 None B C E G H Full survey at [43] Facial Emotion Recognition [3] mini-Xception None C B F G”
BackgroundDhont, J., Wolfs, C. J. A., & Verhaegen, F. (2021) · Medical Physics · doi:10.1002/mp.15419 · Cited by 16
Machine Learning for Multi-Omics Characterization of Blood Cancers: A Systematic Review
Alhamrani, S. Q., Ball, G., El-Sherif, A. A., Ahmed, S. S., Mousa, N. O., Alghorayed, S. A., Alatawi, N. A., Ali, A. M., Alqahtani, F. A., & Gabre, R. M. (2025) · Cells · doi:10.3390/cells14171385 · Cited by 20
“This echoes issues with the development of molecular biomarkers, whereby many fail to transition to clinical use because they were not validated on independent datasets [50].”
Gozzi, N., Giacomello, E., Sollini, M., Kirienko, M., Ammirabile, A., Lanzi, P. L., Loiacono, D., & Chiti, A. (2022) · Diagnostics · doi:10.3390/diagnostics12092084 · Cited by 19
“[44] claimed that none of the works under review was reliable enough for the transition from scientific research to clinical routine due to dataset biases, insufficient model evaluation, limited generalizability, and lack of reproducibility, among other reasons.”
BackgroundRavindran, V., Wagoner, J., Athanasiadis, P., den Hartigh, A. B., Sidorova, J. M., Ianevski, A., Fink, S. L., Frigessi, A., White, J. M., Polyak, S. J., et al. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac456 · Cited by 16
“For instance, so far none of the models for prediction of clinical outcome developed using chest radiographs and CT scans are of potential clinical use, mainly due to methodological flaws and high or unclear risk of bias in the imaging datasets used for the model training [47].”
Artificial Intelligence in Pediatric Endoscopy
Dhaliwal, J., & Walsh, C. M. (2023) · Gastrointestinal Endoscopy Clinics of North America · doi:10.1016/j.giec.2022.12.001 · Cited by 17
A Review on Artificial Intelligence Approaches and Rational Approaches in Drug Discovery
Srivathsa, A. V., Sadashivappa, N. M., Hegde, A. K., Radha, S., Mahesh, A. R., Ammunje, D. N., Sen, D., Theivendren, P., Saravanan, G., Kunjiappan, S., et al. (2023) · Current Pharmaceutical Design · doi:10.2174/1381612829666230428110542 · Cited by 16
Progress and Outlooks in Designing Photonic Biosensor for Virus Detection
Tsalsabila, A., Dabur, V. A., Budiarso, I. J., Wustoni, S., Chen, H., Birowosuto, M. D., Wibowo, A., & Zeng, S. (2024) · Advanced Optical Materials · doi:10.1002/adom.202400849 · Cited by 15
Quickest Detection of COVID-19 Pandemic Onset
Braca, P., Gaglione, D., Maranò, S., Millefiori, L. M., Willett, P., & Pattipati, K. R. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3068072 · Cited by 15
Ma, H., Ye, Q., Ding, W., Jiang, Y., Wang, M., Niu, Z., Zhou, X., Gao, Y., Wang, C., Menpes-Smith, W., et al. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.699984 · Cited by 16
Self-citationGidde, P., Prasad, S. S., Singh, A. P., Bhatheja, N., Prakash, S., Singh, P., Saboo, A., Takhar, R., Gupta, S. K., Saurav, S., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-02003-w · Cited by 16
Imagawa, K., & Shiomoto, K. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105251 · Cited by 15
“These estimated performances may probably be optimistic and misleading because of the high-risk bias based on the non-representative selection of control patients and model overfitting [24,25].”
BackgroundYang, L., Wan, Y., & Pan, F. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01446-1 · Cited by 16
Data Heterogeneity Modeling for Trustworthy Machine Learning
Liu, J., & Cui, P. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736560 · Cited by 17
Bolocan, V.-O., Mitoi, A., Nicu-Canareica, O., Băean, M.-L., Medar, C., & Popa, G.-A. (2025) · Journal of Imaging · doi:10.3390/jimaging11100335 · Cited by 19
“Moreover, competition datasets, while large, often undergo curation that removes problematic cases (motion artifacts, metal implants, incomplete sequences) routinely encountered in clinical practice [43].”
Robust radiomics: a review of guidelines for radiomics in medical imaging
Avanzo, M., Soda, P., Bertolini, M., Bettinelli, A., Rancati, T., Stancanello, J., Rampado, O., Pirrone, G., & Drigo, A. (2026) · Frontiers in Radiology · doi:10.3389/fradi.2025.1701110 · Cited by 19
Framework for Assessing Ethical Aspects of Algorithms and Their Encompassing Socio-Technical System
van Bruxvoort, X., & van Keulen, M. (2021) · Applied Sciences · doi:10.3390/app112311187 · Cited by 16
“For example, many algorithms developed for diagnosis and prognosis of COVID-19 reviewed by [10] were found to be not usable for clinical use, which their developers could have determined themselves if they would have used the CLAIM checklist [46].”
BackgroundMethodReina, A. R., Barrera, J. M., Valdivieso, B., Gas, M.-E., Maté, A., & Trujillo, J. C. (2022) · Scientific Reports · doi:10.1038/s41598-022-09613-y · Cited by 14
Park, S. J., Ye, J. C., Lee, E. S., Cho, G., Yoon, J. W., Choi, J. H., Joo, I., & Lee, Y. J. (2023) · Korean Journal of Radiology · doi:10.3348/kjr.2022.1032 · Cited by 14
“Additionally, the model obtained using supervised learning is prone to bias toward specific image acquisition settings [9].”
Value Proposition of FDA-Approved Artificial Intelligence Algorithms for Neuroimaging
Bajaj, S., Khunte, M., Moily, N. S., Payabvash, S., Wintermark, M., Gandhi, D., & Malhotra, A. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.06.034 · Cited by 14
Arias-Londoño, J. D., & Godino-Llorente, J. I. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105831 · Cited by 18
Systematic Review of Machine Learning Applied to the Secondary Prevention of Ischemic Stroke
Chen, M., Qian, D., Wang, Y., An, J., Meng, K., Xu, S., Liu, S., Sun, M., Li, M., & Pang, C. (2024) · Journal of Medical Systems · doi:10.1007/s10916-023-02020-4 · Cited by 15
“This concentration can also lead to highly similar studies, rather than fostering new insights or breakthroughs [91].”
The curious case of the test set AUROC
Roberts, M. T., Hazan, A., Dittmer, S., Rudd, J. H. F., & Schönlieb, C. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00817-7 · Cited by 17
Self-citationWard, L. T., Blaiszik, B. J., Lee, C., Martin, T., Foster, I. T., & Schleife, A. (2024) · npj Computational Materials · doi:10.1038/s41524-024-01374-8 · Cited by 14
“Here, we demonstrate how the rich data produced by RT-TDDFT simulations can be combined with ML to expand greatly the ability to predict the electronic stopping power of materials, enabling dramatically improved speed while maintaining the accuracy of fi rst principles results.”
Nakayama, L. F., de Matos, J. C. R. G., Quion, J., Novaes, F. D. C., Mitchell, W. G., Mwavu, R., Hung, C. J.-Y. J., Santiago, A. P. D., Phanphruk, W., Cardoso, J. S., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000618 · Cited by 20
“In addition to data and algorithmic bias, other issues in the life cycle include reliance on irrelevant evaluation metrics, and a lack of post-deployment studies [28,35,36].”
BackgroundMethodLi, W., Zhang, Y., Zhou, H., Yang, W., Xie, Z., & He, Y. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103404 · Cited by 15
Artificial intelligence for chest X-ray image enhancement
Song, L., Sun, H., Xiao, H., Lam, S., Zhan, Y., Ren, G., & Cai, J. (2024) · Radiation Medicine and Protection · doi:10.1016/j.radmp.2024.12.003 · Cited by 19
An overview of the National COVID-19 Chest Imaging Database: data quality and cohort analysis
Cushnan, D., Bennett, O., Berka, R., Bertolli, O., Chopra, A., Dorgham, S., Favaro, A., Ganepola, T., Halling‐Brown, M., Imreh, G., et al. (2021) · GigaScience · doi:10.1093/gigascience/giab076 · Cited by 14
“, resolution, projection) rather than disease-related attributes [19].”
Self-citationPrasad, V. K., Bhattacharya, P., Bhavsar, M. D., Verma, A., Tanwar, S., Sharma, G., Bokoro, P. N., & Sharma, R. (2022) · IEEE Access · doi:10.1109/access.2022.3190497 · Cited by 15
“[34] presented a review on the usage of ML approaches to predict computed tomography (CT) image diagnosis, and shows the potential limitations of heterogeneity in gathered images from different hospitals, which limits the accuracy of the models.”
BackgroundMachine learning applications to Covid-19: a state-of-the-art survey
Kamalov, F., Cherukuri, A. K., & Thabtah, F. (2022) · Advances in Science and Engineering Technology International Conferences (ASET) · doi:10.1109/aset53988.2022.9734959 · Cited by 16
“The main obstacle in their deployment is the insufficient sample size which makes the accuracy results unreliable [41].”
BackgroundA Systematic Review and Implementation Guidelines of Multimodal Foundation Models in Medical ImagingPreprint
Huang, S.-C., Jensen, M., Yeung, S. Y., Lungren, M. P., Poon, H., & Chaudhari, A. S. (2025) · Research Square · doi:10.21203/rs.3.rs-5537908/v1 · Cited by 17
“Despite numerous attempts to leverage AI models for healthcare, a signi�cant gap remains between AI’s potential and its current usefulness in clinical practice 1–4 .”
Lee, S. Y., Ha, S., Jeon, M. G., Li, H., Choi, H., Kim, H. P., Choi, Y. R., Hoseok, I., Jeong, Y. J., Park, Y. H., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00658-x · Cited by 13
“While potential utilities of the CADs seem to be promising, recent reviews have found that multiple crucial aspects of model evaluations have been often neglected, especially for commercial CADs 29 – 31 .”
BackgroundInfluentialAnalysis: Flawed Datasets of Monkeypox Skin Images
Vega, C., Schneider, R., & Satagopam, V. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01928-1 · Cited by 15
“The authors concluded that none of the models were adequate for potential clinical use due to methodological flaws and/or underlying biases [4].”
BackgroundDigital Pathology: A Comprehensive Review of Open-Source Histological Segmentation Software
Pavone, A. M., Giannone, A. G., Cabibi, D., D’Aprile, S., Denaro, S., Salvaggio, G., Parenti, R., Yezzi, A. J., & Comelli, A. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4010012 · Cited by 13
“In fact, several studies have been carried out to demonstrate the efficacy of AI-based diagnostic and prognostic models on CT medical images of COVID-19 patients [14–16].”
Empowering brain cancer diagnosis: harnessing artificial intelligence for advanced imaging insights
Al-Kadi, O. S., Al-Emaryeen, R., Al-Nahhas, S., Almallahi, I., Braik, R., & Mahafza, W. S. (2024) · Reviews in the Neurosciences · doi:10.1515/revneuro-2023-0115 · Cited by 13
Aliferis, C., & Simon, G. (2024) · Health informatics · doi:10.1007/978-3-031-39355-6_12 · Cited by 17
Evolving and Novel Applications of Artificial Intelligence in Thoracic Imaging
Chang, J. Y., & Makary, M. S. (2024) · Diagnostics · doi:10.3390/diagnostics14131456 · Cited by 13
“A systematic review by Roberts et al. revealed methodological flaws or a risk of underlying bias in machine learning models for detecting and prognosticating COVID-19 [106].”
BackgroundWu, J. T.-Y., de la Hoz, M. Á. A., Kuo, P., Paguio, J. A., Yao, J. S., Dee, E. C., Yeung, W. L. K., Jurado, J., Moulick, A. N., Milazzo, C., et al. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00674-z · Cited by 13
“1 3 documentation of reproducible methods, (2) adherence to best practices in the development of a model, and (3) external validation that could justify claims of applicability [9].”
MethodSubramanian, M., Easwaramoorthy, S. V., Cho, J., & Shanmugavadivel, K. (2023) · Scientific Reports · doi:10.1038/s41598-023-34908-z · Cited by 14
Cushnan, D., Berka, R., Bertolli, O., Williams, P., Schofield, D., Joshi, I., Favaro, A., Halling‐Brown, M., Imreh, G., Jefferson, E., et al. (2021) · Digital Health · doi:10.1177/20552076211048654 · Cited by 12
“AI tools may improve the accuracy of diagnosis and the estimation of likely prognosis for a patient in some settings, as well as potentially improve efficiencies by speeding up screening processes and reducing the number of human readers required.(7) However, despite data used in AI software development being de-ident…”
BackgroundSelf-citationTer-Sarkisov, A. (2022) · IEEE Intelligent Systems · doi:10.1109/mis.2021.3135474 · Cited by 14
Multimodal Foundation Models for Medical Imaging - A Systematic Review and Implementation GuidelinesPreprint
Huang, S.-C., Jensen, M., Yeung, S. Y., Lungren, M. P., Poon, H., & Chaudhari, A. S. (2024) · medRxiv · doi:10.1101/2024.10.23.24316003 · Cited by 15
The state of the art in artificial intelligence and digital pathology in prostate cancer
Ni, H. M., Kouzy, R., Sabbagh, A., Rooney, M. K., Feng, J., Castillo, S. P., Gadoue, S. M., Kouzi, Z. E., Hoffman, K. E., Yuan, Y., et al. (2025) · Nature Reviews Urology · doi:10.1038/s41585-025-01070-2 · Cited by 13
“A study was conducted to evaluate the clinical potential of published studies investigating the use of AI or ML in detecting or prognosticating COVID-19 with CT scans or plain radiography of the chest 159 .”
Zhang, L., Lei, Z., Xiao, C., Du, Z., Jiang, C., Yuan, X., Hu, Q., Zhai, S., Xu, L., Liu, C., et al. (2022) · Frontiers in Sensors · doi:10.3389/fsens.2022.1015223 · Cited by 13
“Traditional machine learning models such as logistic regression, multilayer perception (Roberts et al., 2021), Support Vector Machine (SVM) (Rendon-Gonzalez and Ponomaryov 2016), K-Nearest Neighbors (KNN) (Nugroho et al.”
Horry, M. J., Chakraborty, S., Pradhan, B., Paul, M., Zhu, J. F., Loh, H. W., Barua, P. D., & Acharya, U. R. (2023) · Sensors · doi:10.3390/s23146585 · Cited by 25
“…that have reported extremely high diagnostic accuracy for COVID-19 pneumonia against “other” viral pneumonias without recognizing that the COVID-19 sample population consisted of elderly patients, while the viral pneumonia control population consisted of pediatric samples, as reported by [37].”
Longitudinal Image Data for Outcome Modeling.
van Timmeren, J. E., Bussink, J., Koopmans, P. J., Smeenk, R., & Monshouwer, R. (2024) · Clinics in oncology · doi:10.1016/j.clon.2024.06.053 · Cited by 13
“Large longitudinal studies have been performed [92], even though longitudinal analyses are not widely explored [93].”
"It depends": Configuring AI to Improve Clinical Usefulness Across Contexts
Zając, H. D., Ribeiro, J., Ingala, S., Gentile, S., Wanjohi, R., Gitau, S. N., Carlsen, J. F., Nielsen, M. B., & Andersen, T. O. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660707 · Cited by 17
“For example, Roberts et al. [61] found that out of the 62 AI models detecting and predicting COVID-19 on chest X-rays and CT scans that were described in the literature, none were deemed to be useful for clinical purposes.”
MethodZhang, M., Zeng, X., Huang, C., Liu, J., Liu, X., Xie, X., & Wang, R. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104545 · Cited by 12
“Because with the huge increased number of AI researches of COVID-19, there were some critical doubt: Michael Roberts et al. [29] systematically reviewed 62 studies about COVID-19 modelling and find that none of the models are of potential clinical use due to methodological flaws and/or underlying…”
BackgroundSunwoo, H., Choi, W., Na, S., Kim, C., & Heo, S. (2022) · Applied Sciences · doi:10.3390/app12063136 · Cited by 12
Dasgupta, A., Bakshi, A., Mukherjee, S., Das, K., Talukdar, S., Chatterjee, P., Mondal, S., Das, P., Ghosh, S., Som, A., et al. (2022) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1462 · Cited by 10
“In this context, Roberts et al. (2021) have investigated the efficacy of several AI models concerning their clinical relevance.”
BackgroundNicol, E. D., Weir-McCall, J. R., Shaw, L. J., & Williamson, E. E. (2022) · Journal of cardiovascular computed tomography · doi:10.1016/j.jcct.2022.07.005 · Cited by 12
Self-citationStarshinova, A. A., Belyaeva, E. N., Doktorovа, N. P., Korotkevich, I., & Kudlay, D. A. (2023) · Journal of Epidemiology and Global Health · doi:10.1007/s44197-023-00085-5 · Cited by 12
“Perhaps, in the coming years, the use of alternative methods of machine learning and artificial intelligence will enable the analysis of computed tomography data, which, given the large number of examinations conducted during the COVID-19 pandemic, may be the most effective [39].”
MethodCOVID-19 Detection from Cough Recordings Using Bag-of-Words Classifiers
Pavel, I., & Ciocoiu, I. B. (2023) · Sensors · doi:10.3390/s23114996 · Cited by 12
Zheng, Y., Huang, D. J., Hao, X., Wei, J., Lu, H., & Liu, Y. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107332 · Cited by 12
Fairness and generalizability of OCT normative databases: a comparative analysis
Nakayama, L. F., Ribeiro, L. Z., de Oliveira, J. A. E., de Matos, J. C. R. G., Mitchell, W. G., Malerbi, F. K., Celi, L. A., & Regatieri, C. V. (2023) · International Journal of Retina and Vitreous · doi:10.1186/s40942-023-00459-8 · Cited by 12
“However, there is a risk of bias when the data, especially that used to determine ‘normal’ parameters, is not representative of the heterogeneous populations for whom the technology is used, with unfair and harmful algorithms outcomes against underrepresented populations [4–6].”
BackgroundJungmann, F., Müller, L., Hähn, F., Weustenfeld, M., Dapper, A.-K., Mähringer‐Kunz, A., Graafen, D., Düber, C., Schafigh, D. G., dos Santos, D. P., et al. (2021) · European Radiology · doi:10.1007/s00330-021-08409-4 · Cited by 11
“However, a systematic review came to an entirely different conclusion and stated that “currently, AI solutions are not of potential clinical use due to underlying methodological weaknesses or biases” [14].”
BackgroundZhu, Y. M., & Falk, T. H. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746471 · Cited by 12
Collaborative learning without sharing data
(2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00364-5 · Cited by 12
A Survey on Data-driven COVID-19 and Future Pandemic Management
Tao, Y., Yang, C., Wang, T., Coltey, E., Jin, Y., Liu, Y., Jiang, R., Fan, Z., Song, X., Shibasaki, R., et al. (2022) · ACM Computing Surveys · doi:10.1145/3542818 · Cited by 12
“For example, it was recently discovered that most of the existing models built for COVID-19 detection using machine learning algorithms for diagnosis cannot be practically used in the clinical setting due to their biases and uncertainties [146].”
BackgroundTopff, L., Lipman, K. B. W. G., Guffens, F., Wittenberg, R., Rutten, A., van Veenendaal, G., Heß, M., Lamerigts, K., Wakkie, J., Ranschaert, E. R., et al. (2023) · European Radiology · doi:10.1007/s00330-022-09303-3 · Cited by 15
“This might especially be worrisome for AI applications in COVID-19 imaging since several methodological flaws and biases in these studies were reported [12].”
BackgroundPerez-Romero, C. A., Mendoza‐Maldonado, L., Tonda, A. P., Coz, E., Tabeling, P., Vanhomwegen, J., MacSharry, J. A., Szafran, J., Bobadilla‐Morales, L., Corona‐Rivera, A., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-42348-y · Cited by 11
Challenges and Potential of Artificial Intelligence in Neuroradiology
Winder, A. J., Stanley, E. A. M., Fiehler, J., & Forkert, N. D. (2024) · Clinical Neuroradiology · doi:10.1007/s00062-024-01382-7 · Cited by 12
“Regardless, systematic reviews suggest that these checklists are underutilized in the current medical AI literature, with a large proportion of model development studies providing insufficient information to motivate follow-up validation and clinical studies [26, 27].”
Nolin-Lapalme, A., Corbin, D., Tastet, O., Avram, R., & Hussin, J. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.04.026 · Cited by 12
Guo, Y., Wang, H., Ren, X., Wang, T., Chen, W., Xu, Z., & Ge, H. (2025) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.70004 · Cited by 11
“In recent years, AI-assisted medicine has demonstrated remarkable promise in various applications such as drug discovery [5, 6], automated diagnostics [7, 8], and pandemic forecasts [9, 10].”
Definitions of necrotizing enterocolitis: What are we defining and is machine learning the answer?
Martin, C. R. (2021) · Pediatric Research · doi:10.1038/s41390-021-01687-0 · Cited by 11
Drozdov, I. A., Szubert, B., Reda, E., Makary, P., Forbes, D., Chang, S. L., Ezhil, A., Puttagunta, S. A., Hall, M., Carlin, C. M., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-99986-3 · Cited by 12
Kuo, M., Chiu, K. W.-H., Wang, D. S., Larici, A. R., Poplavskiy, D., Valentini, A., Napoli, A., Borghesi, A., Ligabue, G., Fang, B., et al. (2022) · European Radiology · doi:10.1007/s00330-022-08969-z · Cited by 11
“…set allowed us to evaluate the model ’ s “ true ” performance across a broad cross section of age groups, disease conditions, localities, and healthcare systems, which is particularly relevant among growing concerns of “ brittle ” AI, model performance overestimations, and cohort bias [29, 30].”
BackgroundVardhan, A., Makhnevich, A., Omprakash, P., Hirschorn, D. S., Barish, M. A., Cohen, S. L., & Zanos, T. P. (2023) · Bioelectronic Medicine · doi:10.1186/s42234-022-00103-0 · Cited by 12
“Although several studies have built models to detect pneumonia, few studies have annotated lung images beyond present/absent labels and do not include information regarding degrees of opacity present (Roberts et al., 2021).”
BackgroundMethodInfluentialArtificial intelligence enabled COVID-19 detection: techniques, challenges and use cases
Chahal, M., Reddy, A., Shah, R., & Shah, J. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15247-7 · Cited by 11
“Since there are fewer COVID-19 chest X-ray images available, training deep learning models from scratch becomes more challenging [32].”
Modelling dataset bias in machine-learned theories of economic decision-making
Thomas, T., Straub, D., Tatai, F., Shene, M., Tosik, T., Kersting, K., & Rothkopf, C. A. (2024) · Nature Human Behaviour · doi:10.1038/s41562-023-01784-6 · Cited by 12
Gourdeau, D., Potvin, O., Biem, J. H., Cloutier, F., Abrougui, L., Archambault, P. M., Chartrand‐Lefebvre, C., Dieumegarde, L., Gagné, C., Gagnon, L., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10136-9 · Cited by 11
Ter-Sarkisov, A. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.108261 · Cited by 11
“According to [34], there is a number of pitfalls that prevent DL models from being used in the real clinical environment.”
Jalalifar, S. A., & Naini, A. S. (2022) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc48229.2022.9871372 · Cited by 12
“…diagnosis, and triage of patients with suspected COVID-19, many previous studies suffer from a lack of standard data collection procedures and performance assessment metrics leading to a high bias and poor generalization, as discussed in details by Roberts et al. [14] and Maguolo et al. [15].”
BackgroundNavigating the development challenges in creating complex data systems
Dittmer, S., Roberts, M. T., Gilbey, J., Biguri, A., SELBY, I. A., Breger, A., Thorpe, M., Weir-McCall, J. R., Gkrania‐Klotsas, E., Korhonen, A., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00665-x · Cited by 10
Self-citationBone age assessment from articular surface and epiphysis using deep neural networks
Deng, Y., Chen, Y., He, Q., Wang, X., Liao, Y., Liu, J., Liu, Z., Huang, J., & Song, T. (2023) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2023585 · Cited by 10
“EfficientNet is an adaptive convolutional neural network, which optimizes the network by searching for the appropriate receptive field, depth, and resolution of convolutional layers [30], and it shows excellent performance on the computer vision tasks [41, 42].”
Machine learning with requirements: A manifesto
Giunchiglia, E., Imrie, F., van der Schaar, M., & Lukasiewicz, T. (2024) · Neurosymbolic Artificial Intelligence · doi:10.3233/nai-240767 · Cited by 10
Non-IID Medical Imaging Data on COVID-19 in the Federated Learning Framework: Impact and Directions
Al-Hafiz, F., & Basuhail, A. A. (2024) · COVID · doi:10.3390/covid4120140 · Cited by 11
Zając, H. D., Andersen, T. O., Kwasa, E., Wanjohi, R., Onyinkwa, M. K., Mwaniki, E. K., Gitau, S. N., Yaseen, S. S., Carlsen, J. F., Fraccaro, M., et al. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3715115 · Cited by 19
“For example, among 62 models aimed to help with the detection and prognostics of COVID-19 on chest radiographs (X-rays) and thoracic Computed Tomography (CT) , not one was identified as clinically useful [106].”
Algorithmic emergence? Epistemic in/justice in AI-directed transformations of healthcare
Emah, I., & Bennett, S. (2025) · Frontiers in Sociology · doi:10.3389/fsoc.2025.1520810 · Cited by 11
Wyatt, M., Hickey, S., Radford, B., González‐Rivero, M., Boutros, N., Callow, N., Ryan, N. M., Chennu, A., Bennamoun, M., & Gilmour, J. (2025) · Ecological Informatics · doi:10.1016/j.ecoinf.2025.103207 · Cited by 10
CALSTER, B. V., van Smeden, M., van Amsterdam, W. A. C., Coemans, M., Wynants, L., & Steyerberg, E. W. (2025) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042324-123749 · Cited by 14
Towards deployment-centric multimodal AI beyond vision and language
Liu, X., Zhang, J., Zhou, S., van der Plas, T. L., Vijayaraghavan, A., Grishina, A., Zhuang, M., Schofield, D., Tomlinson, C., Wang, Y., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01116-5 · Cited by 16
Gillman, A., Lunardo, F., Prinable, J., Belous, G., Nicolson, A., Min, H., Terhorst, A., & Dowling, J. (2021) · Physical and Engineering Sciences in Medicine · doi:10.1007/s13246-021-01093-0 · Cited by 10
“Similar to Roberts et al. [15], we impose a subset of CLAIM, items 7, 9, 20, 21, 22, 25, 26 and 28, as a minimum risk of bias.”
BackgroundMethodInfluentialLiu, F., Chen, D., Zhou, X., Dai, W., & Xu, F. (2022) · Applied Sciences · doi:10.3390/app12083895 · Cited by 11
“On 15 March 2021, a group from the UK [7] published an article pointing out that “our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases”.”
BackgroundAlthenayan, A. S., Alsalamah, S., Aly, S. F., Nouh, T. A., & Mirza, A. A. (2022) · Applied Sciences · doi:10.3390/app122010535 · Cited by 11
Starshinova, A. A., Osipov, N. N., Dovgalyk, I., Kulpina, A., Belyaeva, E. N., & Kudlay, D. A. (2024) · Diagnostics · doi:10.3390/diagnostics14070698 · Cited by 11
“Some researchers draw attention to the need for wider use of MRI research, especially for the detection and diagnosis of extrapulmonary TB [55,56].”
BackgroundReal-World Surveillance of FDA-Cleared Artificial Intelligence Models: Rationale and Logistics
Dreyer, K. J., Allen, B., & Wald, C. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.06.025 · Cited by 10
Li, M., Chang, K., Mei, X., Bernheim, A., Chung, M., Steinberger, S., Kalpathy–Cramer, J., & Little, B. P. (2021) · American Journal of Roentgenology · doi:10.2214/ajr.21.26717 · Cited by 10
Helbing, D., Beschorner, T., Frey, B. S., Diekmann, A., Hagendorff, T., Seele, P., Spiekermann, S., van den Hoven, J., & Zwitter, A. (2021) · Journal of European CME · doi:10.1080/21614083.2021.1989243 · Cited by 10
“Other studies have yielded similar daunting results [17,18].”
ResultTricarico, D., Calandri, M., Barba, M., Piatti, C., Geninatti, C., Basile, D., Gatti, M., Melis, M., & Veltri, A. (2022) · Diagnostics · doi:10.3390/diagnostics12030570 · Cited by 10
“However methodological flaws and underlying bias are common among these papers, as recently highlighted [30].”
Background
Metadata
- Authors
- Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., Weir-McCall, J. R., Teng, Z., Gkrania-Klotsas, E., AIX-COVNET, Ruggiero, A., Korhonen, A., Jefferson, E., Ako, E., Langs, G., Gozaliasl, G., Yang, G., Prosch, H., Preller, J., Stanczuk, J., Tang, J., Hofmanninger, J., Babar, J., Sánchez, L. E., Thillai, M., Gonzalez, P. M., Teare, P., Zhu, X., Patel, M., Cafolla, C., Azadbakht, H., Jacob, J., Lowe, J., Zhang, K., Bradley, K., Wassin, M., Holzer, M., Ji, K., Ortet, M. D., Ai, T., Walton, N., Lio, P., Stranks, S., Shadbahr, T., Lin, W., Zha, Y., Niu, Z., Rudd, J. H. F., Sala, E., & Schönlieb, C.-B.
- Affiliations
- AstraZeneca (United Kingdom) (GB)
- University of Cambridge (GB)
- University of Manchester (GB)
- Royal Papworth Hospital (GB)
- University of Dundee (GB)
- Royal Brompton Hospital (GB)
- Medical University of Vienna (AT)
- University of Helsinki (FI)
- Imperial College London (GB)
- Boehringer Ingelheim (Germany) (DE)
- Addenbrooke's Hospital (GB)
- University College London (GB)
- Macau University of Science and Technology (MO)
- Tongji Hospital (CN)
- Renmin Hospital of Wuhan University (CN)
- Corresponding author
- Roberts, Michael
- ORCID
- Roberts, Michael
- Thorpe, Matthew
- Gilbey, Julian
- Yeung, Michael
- Ursprung, Stephan
- McCague, Cathal
- Weir-McCall, Jonathan R.
- Gkrania-Klotsas, Effrossyni
- Ako, Emmanuel
- Langs, Georg
- Gozaliasl, Ghassem
- Yang, Guang
- Prosch, Helmut
- Preller, Jacobus
- Azadbakht, Hojjat
- Jacob, Joseph
- Ji, Kangyu
- Walton, Nicholas
- Stranks, Samuel
- Shadbahr, Tolou
- Rudd, James H. F.
- Sala, Evis
- Schönlieb, Carola-Bibiane
- Venue
- Nature Machine Intelligence
- Published
- 2021-03-15
- Volume
- 3
- Issue
- 3
- Pages
- 199-217
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W3119527628
- Cited by
- 1,044
- References
- 83
- Citations by year
- 2026: 203 · 2025: 96 · 2024: 143 · 2023: 248 · 2022: 224 · 2021: 130
- FWCI
- 98.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Machine Intelligence (publishedVersion, cc-by)
- Discovery Research Portal (University of Dundee) (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- elib (German Aerospace Center) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Työväentutkimus Vuosikirja (submittedVersion)
- arXiv (Cornell University)
- Funders
- AstraZeneca
- European Commission (777826)
- Medical Research Council (1966157, MR/S010351/1)
- Engineering and Physical Sciences Research Council (EP/T017961/1, EP/N014588/1, EP/S026045/1)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- COVID-19 pandemic, machine learning, chest radiograph, chest CT, COVID-19 detection, outcome prediction, clinical utility, model validation, methodological errors, bias, research recommendations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
