Methodologic Guide for Evaluating Clinical Performance and Effect of Artificial Intelligence Technology for Medical Diagnosis and Prediction
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
The use of artificial intelligence in medicine is currently an issue of great interest, especially with regard to the diagnostic or predictive analysis of medical images. Adoption of an artificial intelligence tool in clinical practice requires careful confirmation of its clinical utility. Herein, the authors explain key methodology points involved in a clinical evaluation of artificial intelligence technology for use in medicine, especially high-dimensional or overparameterized diagnostic or predictive models in which artificial deep neural networks are used, mainly from the standpoints of clinical epidemiology and biostatistics. First, statistical methods for assessing the discrimination and calibration performances of a diagnostic or predictive model are summarized. Next, the effects of disease manifestation spectrum and disease prevalence on the performance results are explained, followed by a discussion of the difference between evaluating the performance with use of internal and external datasets, the importance of using an adequate external dataset obtained from a well-defined clinical cohort to avoid overestimating the clinical performance as a result of overfitting in high-dimensional or overparameterized classification model and spectrum bias, and the essentials for achieving a more robust clinical evaluation. Finally, the authors review the role of clinical trials and observational outcome studies for ultimate clinical verification of diagnostic or predictive artificial intelligence tools through patient outcomes, beyond performance metrics, and how to design such studies. © RSNA, 2018
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This common scenario means that estimations of AI performance based on internal test data may overestimate its real-world performance on external data [73–75].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [74]
In the citing paper
“Similar to internal validation of the AIPM, external AIPM model validation can be based on discrimination (area under the receiver operating characteristic curve, sensitivity, specificity, positive and negative predictive values), calibration (calibration plot)5,6,13,17,49,56,78, and Decision Curve Analysis5,6,17,49,68,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“To ensure the generalizability of the AIPM to the intended healthcare setting, developers are advised to extensively validate the model for representative data from that setting6–8,10,13–15,24,27,56,64,66,68,72,77,79,87,88 (see phase 1, Representativeness).”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“The literature recommends to assess generalizability on external data from a different time period, place, or healthcare setting3,7,8,10,16,17,27,56,68,72,79.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“Although prospective validation is rare, it is preferred by the literature5,14,56, as it provides a better idea of the AIPM’s true applicability to medical practice and allows the healthcare professionals to identify and review errors in real time19,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
(2016) · Cited by 11,679
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293
Lakhani, P., & Sundaram, B. (2017) · Radiology · doi:10.1148/radiol.2017162326 · Cited by 1,775
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Radiology · doi:10.1148/radiol.2015151516 · Cited by 1,337
Deep Learning in Medical Imaging: General Overview
Lee, J.-G., Jun, S., Cho, Y.-W., Lee, H., Kim, G. B., Seo, J. B., & Kim, N. (2017) · Korean Journal of Radiology · doi:10.3348/kjr.2017.18.4.570 · Cited by 1,276
Research works printed but not linked: 3
- Steyerberg EW, 2010, Clinical prediction models: a practical approach to development, validation, and updating, 83
- Steyerberg EW, 2010, Clinical prediction models: a practical approach to development, validation, and updating, 299
- Newman TB, 2013, Designing clinical research, 117
Cited by
Items 401–500 of 1,000
Chervenkov, L., Sirakov, N., Kostov, G., Velikova, T. V., & Hadjidekov, G. (2023) · World Journal of Radiology · doi:10.4329/wjr.v15.i5.136 · Cited by 19
Moon, S. J., Lee, S., Hwang, J., Lee, J., Kang, S., & Cha, H. (2023) · RMD Open · doi:10.1136/rmdopen-2023-003783 · Cited by 19
“11 External validation involves testing the model in a completely independent cohort.”
InfluentialOrtiz, A., & Mulsant, B. H. (2024) · Journal of Medical Internet Research · doi:10.2196/59826 · Cited by 19
“These challenges include unreliable inherent assumptions [31,32], model instability [33], and lack of interpretability [34] or explainability [35] of results (the black box problem).”
Diniz, J. B. C., Santana, L. S., Leite, M. G. H. S. J., Santana, J. L. S., Costa, S. I. M., Castro, L. H. M., & Telles, J. P. M. (2024) · Seizure · doi:10.1016/j.seizure.2024.09.019 · Cited by 19
Shah, H. H., & Bacha, A. (2025) · Global journal of universal studies. · doi:10.70445/gjus.1.2.2024.87-107 · Cited by 19
“Despite these ongoing challenges, efforts continue around explainable AI (XAI) techniques to make the inner workings of AI models more transparent and interpretable [56].”
The optimization of software testing efficiency and effectiveness using AI techniques
Talakola, S. (2024) · International Journal of Artificial Intelligence Data Science and Machine Learning · doi:10.63282/3050-9262.ijaidsml-v5i3p104 · Cited by 19
Bae, H., Lee, H., Kim, S., Han, K., Rhee, H., Kim, D. K., Kwon, H. P., Hong, H., & Lim, J. S. (2021) · European Radiology · doi:10.1007/s00330-021-07877-y · Cited by 19
“However, due to the retrospective nature of these studies, researchers tended to adopt convenience sampling, the data collection method of sampling control, and diseased subjects in a calculated ratio that can cause spectrum bias [12, 13].”
BackgroundSelf-citationLiew, C. J. Y., Krishnaswamy, P., Cheng, L. T., Tan, C. H., Poh, A. C. C., & Lim, C. C. T. (2019) · Annals of the Academy of Medicine Singapore · doi:10.47102/annals-acadmedsg.v48n1p16 · Cited by 19
Raeisi, Z., Rokhva, S., Rahmani, F., Goodarzi, A., & Najaf-Zadeh, H. (2025) · Oral and Maxillofacial Surgery · doi:10.1007/s10006-025-01463-y · Cited by 18
Hack, S., Alsleibi, S., Shemesh, S., & Nakache, G. (2025) · SLEEP · doi:10.1093/sleep/zsaf338 · Cited by 18
Developing and Implementing a Web-Based Platform for Skills and Knowledge Exchange
Alghamdi, A. K. H., Harbi, D. A., Alarfaj, N., Hajri, B. A., Almarashdeh, I., Alsmadi, M. K., Alshabanah, M., & Alrajhi, D. (2019) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst196345 · Cited by 18
Wu, R., Liu, H., Li, H., Chen, L., Wei, L., Huang, X., Liu, X., Men, X., Li, X., Han, L., et al. (2023) · BioMedical Engineering OnLine · doi:10.1186/s12938-023-01164-1 · Cited by 18
“SWS of patients from SYSUTH (a hospital in Guangzhou city) and MMPH (a hospital in Maoming city) served as the development set, and SWS of patients from STUMFH (a hospital in Shan-tou city) served as the external test set for testing the geographic performance of the model [33].”
MethodAdvances in radiology and pathology of prostate cancer: a review for the pathologist
Pepe, P., Fandella, A., Barbera, M., Martino, P., Merolla, F., Caputo, A., & Fraggetta, F. (2024) · Pathologica · doi:10.32074/1591-951x-925 · Cited by 18
“In recent years, significant advances in artificial intelligence and computer-aided diagnosis have improved the diagnosis of PCa, both radiologically and histologically 79,80 .”
BackgroundLiu, Z., Tieu, A., Patel, N., Soultanidis, G., Deyer, L., Wang, Y., Huver, S. D., Zhou, A., Mei, Y., Fayad, Z. A., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73290-4_10 · Cited by 18
“This is further compounded by an overall lack of high-quality and open-source medical imaging data [6–8], hindering the generalizability and accuracy of developed models [9, 10].”
Mazin, A., Hawkins, S. H., Stringfield, O., Dhillon, J., Manley, B. J., Jeong, D. K., & Raghunand, N. (2021) · Scientific Reports · doi:10.1038/s41598-021-83271-4 · Cited by 18
Auditor Judgment Bias Research: A 50-Year Trend Analysis and Emerging Technology Use
Lombardi, D. R., Sipior, J. C., & Dannemiller, S. (2023) · Journal of Information Systems · doi:10.2308/isys-2020-079 · Cited by 18
Walczak, S. (2018) · International Journal of Computers in Clinical Practice · doi:10.4018/ijccp.2018070103 · Cited by 18
“There is a tremendous need for incorporating artificial intelligence into clinical decision making to improve clinical outcomes (Park & Han, 2018).”
BackgroundDevelopment and Comparison of Machine Learning Algorithms to Determine Visual Field Progression
Saeedi, O. J., Boland, M. V., D’Acunto, L., Swamy, R. N., Hegde, V., Gupta, S., Venjara, A., Tsai, J., Myers, J. S., Wellik, S. R., et al. (2021) · Translational Vision Science & Technology · doi:10.1167/tvst.10.7.27 · Cited by 18
COVID-19 prediction models should adhere to methodological and reporting standards
Collins, G. S., van Smeden, M., & Riley, R. D. (2020) · European Respiratory Journal · doi:10.1183/13993003.02643-2020 · Cited by 18
Cheng, D., Khaw, C. R., McCabe, J., Pennycuick, A., Nair, A., Moore, D. A., Janes, S. M., & Jacob, J. (2024) · Clinical Radiology · doi:10.1016/j.crad.2024.04.022 · Cited by 17
Impact of metadata in multimodal classification of bone tumours
Hinterwimmer, F., Guenther, M., Consalvo, S., Neumann, J., Gersing, A. S., Woertler, K., von Eisenhart‐Rothe, R., Burgkart, R., & Rueckert, D. (2024) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-024-07934-9 · Cited by 17
“This assessment is crucial before considering the model’s suitability for clinical use [32].”
MethodArtificial intelligence: a new era in prostate cancer diagnosis and treatment
Vidiyala, N., Parupathi, P. R., Sunkishala, P., Muppavarapu, C. S., Gujja, A., Kanagala, P., Meduri, S. K., & Nyavanandi, D. (2025) · International Journal of Pharmaceutics · doi:10.1016/j.ijpharm.2025.126024 · Cited by 17
Confidence Calibration: An Introduction With Application to Quality Improvement.
Amini, B., Bassett, R., Haygood, T., McEnery, K., & Richardson, M. L. (2020) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2019.12.009 · Cited by 13
Park, J. E., Kim, H. S., Lee, J., Cheong, E., Shin, I., Ahn, S. S., & Shim, W. H. (2020) · Scientific Reports · doi:10.1038/s41598-020-78485-x · Cited by 17
Kim, H., Lee, J. H., Kim, H. J., Park, C. M., Wu, H., & Goo, J. M. (2021) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2021.10.022 · Cited by 17
Diabetic wounds and artificial intelligence: A mini-review
Tehsin, S., Kausar, S., & Jameel, A. (2023) · World Journal of Clinical Cases · doi:10.12998/wjcc.v11.i1.84 · Cited by 17
Human versus machine in medicine: can scientific literature answer the question?
Cook, T. S. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30124-4 · Cited by 17
Li, Z., Dong, A., & Zhou, J. (2020) · Computational and Mathematical Methods in Medicine · doi:10.1155/2020/5294840 · Cited by 17
Kim, R. Y. (2024) · Cancer Biomarkers · doi:10.3233/cbm-230360 · Cited by 16
de Camargo, T. F. O., Ribeiro, G., da Silva, M. C. B., da Silva, L. O., Torres, P. P. T. E. S., Silva, D. D., de Santos, M. O. N., Salibe-Filho, W., Rosa, M. E. E., Novaes, M. D. A., et al. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1512910 · Cited by 16
Fujita, M., Nagashima, K., Takahashi, S., Suzuki, K., Fujisawa, T., & Hata, A. (2019) · Respirology · doi:10.1111/resp.13602 · Cited by 16
Using Artificial Intelligence in Fungal Lung Disease: CPA CT Imaging as an Example
Angelini, E. D., & Shah, A. (2021) · Mycopathologia · doi:10.1007/s11046-021-00546-0 · Cited by 16
Kim, U.-H., Kim, M. Y., Park, E., Lee, W., Lim, W., Kim, H., Oh, S., & Jin, K. N. (2021) · Korean Journal of Radiology · doi:10.3348/kjr.2021.0201 · Cited by 16
“Further multicenter research with data augmentation and temporal and geographic validation is needed to overcome these gaps [31].”
BackgroundChoi, J. W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-09002-8_44 · Cited by 16
Analysis of Robotics, Artificial intelligence and Machine learning in the field of healthcare sector
Boddu, R. S. K., Ahamad, S., Kumar, K. V. P., Ramalingam, M., Pallathadka, L. K., & Tupas, F. P. (2021) · Materials Today Proceedings · doi:10.1016/j.matpr.2021.12.016 · Cited by 16
Artificial Intelligence for Breast US
Villa‐Camacho, J. C., Baikpour, M., & Chou, S.-H. S. (2022) · Journal of Breast Imaging · doi:10.1093/jbi/wbac077 · Cited by 16
Artificial intelligence predicts lung cancer radiotherapy response: A meta-analysis
Xing, W., Gao, W., Lv, X., Zhao, Z., Xu, X., Wu, Z., Mao, G., & Chen, J. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102585 · Cited by 16
Osapoetra, L. O., Sannachi, L., Quiaoit, K., Dasgupta, A., DiCenzo, D., Fatima, K., Wright, F. C., Dinniwell, R. E., Trudeau, M., Gandhi, S., et al. (2021) · Oncotarget · doi:10.18632/oncotarget.27867 · Cited by 16
“In contrast to previous studies, the work here improved study design by limiting the number of selected features in developing a response predictive model in order to prevent overfitting [50].”
BackgroundMethodResultInfluentialMah, W. L. W., Kumar, D., Jin, T., & Piramanayagam, S. N. (2021) · Journal of Magnetism and Magnetic Materials · doi:10.1016/j.jmmm.2021.168131 · Cited by 16
Lee, S., Jung, J., Nam, Y., Lee, S. Y., Park, H., Shin, S., Chung, Y., & Jung, C. K. (2021) · Scientific Reports · doi:10.1038/s41598-021-94826-w · Cited by 16
Provenzano, D., Melnyk, O., Imtiaz, D., McSweeney, B., Nemirovsky, D., Wynne, M., Whalen, M. J., Rao, Y. J., Loew, M. H., & Haji-Momenian, S. (2023) · Applied Sciences · doi:10.3390/app13021088 · Cited by 15
Artificial Intelligence Diagnosis of Parkinson's Disease From MRI Scans
Reddy, S., Giri, D., & Patel, R. (2024) · Cureus · doi:10.7759/cureus.58841 · Cited by 15
“Beyond imaging, AI technologies are increasingly utilized in the realm of precision medicine, facilitating the customization of treatment plans based on individual patient characteristics, genetic profiles, and biomarker signatures [16].”
Evolving and Novel Applications of Artificial Intelligence in Thoracic Imaging
Chang, J. Y., & Makary, M. S. (2024) · Diagnostics · doi:10.3390/diagnostics14131456 · Cited by 15
“Demonstrating the clinical utility of AI through more clinical trials and observational studies is imperative, as improved diagnostic accuracy may not always translate to better patient outcomes [14,153].”
BackgroundEbrahimi, R., Rahmani, M., Fallahtafti, P., Raeini, A. G., Azarboo, A., Jalali, A., & Mehrani, M. (2024) · Therapeutic Advances in Cardiovascular Disease · doi:10.1177/17539447241290438 · Cited by 15
“52 On the other hand, external validation tests the performance of the model in another population and is of utmost importance for the assessment of model generalizability.”
Kemper, E. H. M., Erenstein, H., Boverhof, B.-J., Redekop, K., Андрейченко, А. Е., Dietzel, M., Lipman, K. B. W. G., Huisman, M., Klontzas, M. E., Vos, F. M., et al. (2024) · European Radiology · doi:10.1007/s00330-024-11188-3 · Cited by 15
Altom, D. S., Taha, A. I. A., Hussein, A. A. A. M., Elshiekh, M. A. I., Abdelmajed, A. H., Ibrahim, F. I. A., Mohammed, S., & Tifoor, M. M. E. E. (2025) · Cureus · doi:10.7759/cureus.81001 · Cited by 16
“A growing number of conversational agents are able to provide genuine interactions between people and their artificially intelligent agent counterparts thanks to recent developments in AI software [27].”
Joo, B. (2025) · Neurointervention · doi:10.5469/neuroint.2025.00283 · Cited by 15
Next-Generation Advances in Prostate Cancer Imaging and Artificial Intelligence Applications
Miao, K. H., Miao, J. H., Finkelstein, M., Chatterjee, A., & Oto, A. (2025) · Journal of Imaging · doi:10.3390/jimaging11110390 · Cited by 15
“Through such concerted efforts, AI can assist clinicians in prostate cancer care, delivering precision diagnostics and personalized therapies that ultimately improve survival and quality of life for patients [59,62].”
Yildirim, A., Hertach, R., & Yildirim, V. (2025) · Journal of Medicine and Dentistry · doi:10.64951/jmdnt.2026.1.08 · Cited by 15
Hwang, E. J., Kim, H., Lee, J. H., Goo, J. M., & Park, C. M. (2020) · European Radiology · doi:10.1007/s00330-020-07062-7 · Cited by 15
“Many previous investigations on DL-based models have demonstrated excellent discrimination; however, calibration is often overlooked, and the calibration of DL-based models has rarely been investigated [22].”
BackgroundArtificial Intelligence-related Literature in Transplantation: A Practical Guide
Park, S. H., Mazumder, N. R., Mehrotra, S., Ho, B., Kaplan, B. E., & Ladner, D. P. (2020) · Transplantation · doi:10.1097/tp.0000000000003304 · Cited by 15
“The AUROC curve indicates the relationship between false-positive rate (1-specificity) and true-positive rate (sensitivity) 6 (Figure 1C).”
BackgroundSelf-citationBao, C., Shen, J., Zhang, Y., Zhang, Y., Wei, W., Wang, Z., Ding, J., & Han, L. (2022) · Cancer Medicine · doi:10.1002/cam4.5231 · Cited by 15
Chen, M., Liang, X., & Xu, Y. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/5993839 · Cited by 15
“In recent years, with the continuous in-depth research and rapid development as well as the application of expert systems in the field of medical diagnosis, a good foundation has been laid for the development of intelligent mental consultation [20].”
BackgroundHuang, W., Li, Y., Bao, Z., Ye, J., Xia, W., Lv, Y., Lü, J., Wang, C., & Zhu, X. (2024) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s451301 · Cited by 15
Colonoscopy and artificial intelligence: Bridging the gap or a gap needing to be bridged?
Ang, T. L., & Li, J. W. (2021) · Artificial Intelligence in Gastrointestinal Endoscopy · doi:10.37126/aige.v2.i2.36 · Cited by 15
Introduction to artificial intelligence for cardiovascular clinicians
Chang, A. C., & Limon, A. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-90534-3.00010-x · Cited by 14
Mulliqi, N., Blilie, A., Ji, X., Szolnoky, K., Olsson, H., Titus, M., Gonzalez, G. M., Boman, S. E., Valkonen, M., Gudlaugsson, E. G., et al. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-097591 · Cited by 14
“34 To our knowledge, the studies outlined in this protocol represent the largest retrospective validation of an AI system for prostate cancer diagnosis using digitised core needle biopsies.”
Fang, Z., Johnston, A., Cheuy, L. Y., Na, H. S., Paschali, M., González, C., Armstrong, B. A., Koirala, A., Laurel, D., Campion, A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02007-0 · Cited by 14
“Since AI model performance is known to vary with disease prevalence 55,56 , we evaluated both the primary model and EMM performance across different prevalence levels.”
Evaluating the Performance and Clinical Utility of AI-driven Diagnostic Tools in Radiology
Mercaldo, S., Hillis, J. M., & Blume, J. D. (2025) · Radiology · doi:10.1148/radiol.243935 · Cited by 14
Developing and Implementing an Online Learning Platform for Children with Autism
Dawodi, A. F. A., Alzahrani, S. F., Almumtin, R. A., Alshyban, S. S., Alshabanah, M., Alrajhi, D., Alsmadi, M. K., & Almarashdeh, I. (2020) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst207162 · Cited by 14
Review of Artificial Intelligence Training Tools and Courses for Radiologists
Richardson, M. L., Adams, S., Agarwal, A. K., Auffermann, W. F., Bhattacharya, A. K., Consul, N., Fotos, J. S., Kelahan, L. C., Lin, C. S., Lo, H. S., et al. (2021) · Academic Radiology · doi:10.1016/j.acra.2020.12.026 · Cited by 15
ORAL VE MAKSİLLOFASİYAL RADYOLOJİ’DE YAPAY ZEKA
ÖZKESİCİ, M. Y., & Yılmaz, S. (2021) · Sağlık Bilimleri Dergisi · doi:10.34108/eujhs.1040476 · Cited by 14
Wang, J., Su, G., Wan, C., Huang, X., & Sun, L. (2020) · Symmetry · doi:10.3390/sym12060903 · Cited by 14
“Artificial intelligence (AI), Internet of Things (IoT) and big data are helping radiologists to develop deep neural networks for classification, detection, and segmentation tasks of different diseases that are threatening the health of millions of patients [5,6].”
BackgroundConvolutional neural network for biomedical applications
Abraham, G. K., Jayanthi, V. S., & Bhaskaran, P. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-820604-1.00010-8 · Cited by 14
Xing, M., Zhang, Y., Yu, H., Yang, Z., Li, X., Li, Q., Zhao, Y., Zhao, Z., & Luo, Y. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107103 · Cited by 14
Rethinking Machine Learning Model Evaluation in PathologyPreprint
Javed, S. A., Juyal, D., Shanis, Z., Chakraborty, S., Pokkalla, H., & Prakash, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.05205 · Cited by 14
Developing and Implementing A System for Shipping Companies Comparison
Al-Theeb, R., Al-Tami, H., Al-Johani, H., Al-Mutairi, A., Almarashdeh, I., Alsmadi, M. K., Alshabanah, M., & Alrajhi, D. (2019) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst19649 · Cited by 13
Artificial Intelligence Based Decision Support System for Education Management in Higher Education
Widodo, Y. B., Korwa, R. U., & Nuraini, R. (2023) · al-fikrah Jurnal Manajemen Pendidikan · doi:10.31958/jaf.v11i2.12120 · Cited by 13
Delfan, N., Abbası, F., Emamzadeh, N., Bahri, A., Parvaresh, M., Motamedi, A., Moshiri, B., & Iranmehr, A. (2025) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2025.111243 · Cited by 13
Hwang, E. J., Lee, J. H., Kim, J. H., Lim, W. H., Goo, J. M., & Park, C. M. (2021) · BMC Pulmonary Medicine · doi:10.1186/s12890-021-01768-0 · Cited by 13
“Calibration of a deep learning-based algorithm is an important factor for its interpretability and credibility, but this aspect is frequently overlooked [27].”
BackgroundArtificial neural network in the discrimination of lung cancer based on infrared spectroscopy
Lugtu, E. J., Ramos, D. B., Agpalza, A. J., Cabral, E. A., Carandang, R. P., Dee, J. E., Martinez, A., Jose, J. E., Santillan, A., Bangaoil, R., et al. (2022) · PLoS ONE · doi:10.1371/journal.pone.0268329 · Cited by 13
“This study utilized a diagnostic case-control research design which creates spectrum biases due to limited samples, ergo the diagnostic performance may be inflated [93, 94].”
MethodThyroid Nodule Malignancy Risk Stratification Using a Convolutional Neural Network
Stib, M. T., Pan, I., Merck, D. L., Middleton, W. D., & Beland, M. D. (2020) · Ultrasound Quarterly · doi:10.1097/ruq.0000000000000501 · Cited by 13
Yang, Y., Tang, Y., Gao, R., Bao, S., Huo, Y., McKenna, M. T., Savona, M. R., Abramson, R. G., & Landman, B. A. (2021) · Journal of medical imaging · doi:10.1117/1.jmi.8.1.014004 · Cited by 13
“10 Given an image from ultrasound, magnetic resonance imaging, or computed tomography (CT), manual annotation of spleen segmentation is still the gold standard, 11 yet the process is time-intensive and requires domain expertise.”
BackgroundButryn, B., Chomiak-Orsa, I., Hauke, K., Pondel, M., & Siennicka, A. E. (2021) · Procedia Computer Science · doi:10.1016/j.procs.2021.09.086 · Cited by 13
Bartoli, A., Fournel, J., Maurin, A., Marchi, B., Habert, P., Castelli, M., Gaubert, J., Cortaredona, S., Lagier, J., Million, M., et al. (2022) · Research in Diagnostic and Interventional Imaging · doi:10.1016/j.redii.2022.100003 · Cited by 13
“The MVSF results are presented in Appendix C.”
MethodInfluentialQuantifying uncertainty in machine learning classifiers for medical imaging
Valen, J., Balki, I., Mendez, M., Qu, W., Levman, J. E. D., Bilbily, A., & Tyrrell, P. N. (2022) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-022-02578-3 · Cited by 13
“…post-test probability of a classifier (i.e., the predictive probability for a particular test image) is very dependent on the pre-test probability (i.e., the disease prevalence in the data) if the sample size is small, or if the likelihood ratio (LR) is small (from a poorly suited algo-rithm) [5].”
BackgroundDeep Learning for Triage of Chest Radiographs: Should Every Institution Train Its Own System?
van Ginneken, B. (2018) · Radiology · doi:10.1148/radiol.2018182318 · Cited by 13
Decoding ChatGPT’s ‘impact’ on the future of healthcare
Pearce, H. A., & Roop, P. S. (2023) · Cancer Research Statistics and Treatment · doi:10.4103/crst.crst_84_23 · Cited by 12
Recommender-based bone tumour classification with radiographs—a link to the past
Hinterwimmer, F., Serena, R. S., Wilhelm, N., Breden, S., Consalvo, S., Seidl, F., Jüstel, D., Burgkart, R., Woertler, K., von Eisenhart‐Rothe, R., et al. (2024) · European Radiology · doi:10.1007/s00330-024-10672-0 · Cited by 12
“…set consists of external radiographs from general practitioners, external radiologists, etc. uploaded to our clinical systems, another limitation is that the model needs to be tested on external data to further assess gen-eralisability [32] before suitability for clinical use can be evaluated [33].”
BackgroundArtificial intelligence‐driven change redefining radiology through interdisciplinary innovation
Huang, R., Meng, X., Zhang, X., Luo, Z., Cao, L., Feng, Q., Ma, G., Dong, D., & Wang, Y. (2025) · Interdisciplinary medicine · doi:10.1002/inmd.20240063 · Cited by 13
Saatçi, E. Y. (2025) · International Journal of Economic Behavior and Organization · doi:10.11648/j.ijebo.20251301.14 · Cited by 12
Jing, Y., Duan, Z., Gao, J., Chen, S., Bing, Z., & Wang, Y. (2022) · Frontiers in Energy Research · doi:10.3389/fenrg.2022.943213 · Cited by 12
“In terms of application results, it has shown good application effects (Sutton and Barto, 1998; Leo Kumar, 2017; Park and Han, 2018).”
BackgroundLee, S. B., Hong, Y., Cho, Y. J., Jeong, D., Lee, J., Yoon, S. H., Lee, S., Choi, Y. H., & Cheon, J. (2023) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0588 · Cited by 12
“Large and heterogeneous datasets with high-quality images from multiple institutions and different geographic areas are essential for training and developing high-quality deep learning algorithms with general applications [8].”
BackgroundGomes, R. S. A., de Oliveira, G. H. P., de Moura, D. T. H., Kotinda, A. P. S. T., Matsubayashi, C. O., Hirsch, B. S., Veras, M. D. O., Sasso, J. G. R. J., Trasolini, R. P., Bernardo, W. M., et al. (2023) · World Journal of Gastrointestinal Endoscopy · doi:10.4253/wjge.v15.i8.528 · Cited by 12
Gonzales, R. A., Ibañez, D., Hann, E., Popescu, I. A., Burrage, M. K., Lee, Y. P., Altun, İ., Weintraub, W. S., Kwong, R. Y. K., Kramer, C. M., et al. (2023) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2023.1213290 · Cited by 12
“5% for the test dataset (309 LGE/VNE images from 103 patients), per recommended guidelines (36).”
BackgroundZhang, Y., Jing, R., Ji, X., & Hu, N. (2023) · Open Computer Science · doi:10.1515/comp-2022-0280 · Cited by 12
Zhou, B., Ai, W., & Cao, H. (2023) · International Journal of Orthopaedic and Trauma Nursing · doi:10.1016/j.ijotn.2023.101077 · Cited by 13
Huang, J., Zhu, X., Ma, Y., Zhang, Z., Zhang, J., Zhou, H., Wu, L., Liu, H., Wu, H., & Bao, C. (2025) · Translational Gastroenterology and Hepatology · doi:10.21037/tgh-24-117 · Cited by 12
“Furthermore, selection bias can easily occur if patient selection is not random or continuous, whereby the study results cannot be adequately extrapolated to the target population in clinical practice (77,82,83).”
An Adaptive Deep Ensemble Learning Method for Dynamic Evolving Diagnostic Task Scenarios
Su, K., Wu, J., Gu, D., Yang, S., Deng, S., & Khakimova, A. K. (2021) · Diagnostics · doi:10.3390/diagnostics11122288 · Cited by 12
“AUC is the area under the ROC curve, and it is the most commonly used summary measure of a ROC curve [52].”
MethodCayot, B., Milot, L. M., Nempont, O., Vlachomitrou, A. S., Langlois, C., Dumortier, J., Boillot, O., Arnaud, K., Barten, T. R. M., Drenth, J. P. H., et al. (2022) · European Radiology · doi:10.1007/s00330-022-08549-1 · Cited by 12
“To develop our CNN and evaluate its performance, three separate phases were completed: a training phase to train the network, a validation phase to optimize the final network by iteration, and a test phase to evaluate the performance of the final model [25].”
MethodBoukobza, A., Burgun, A., Roudier, B., & Tsopra, R. (2022) · JMIR Medical Informatics · doi:10.2196/34306 · Cited by 12
“First, our method was developed on a data set of tweets in English and needs to be adapted for other languages [72] and assessed with other extensive data sets [49,73].”
MethodAutomated detection of smiles as discrete episodes
Mohammed, H., Kumar, R., Bennani, H., Halberstadt, J. B., & Farella, M. (2022) · Journal of Oral Rehabilitation · doi:10.1111/joor.13378 · Cited by 12
Artificial Intelligence in Radiology: Resident Recruitment Help or Hindrance?
Grayev, A. M. (2019) · Academic Radiology · doi:10.1016/j.acra.2019.01.005 · Cited by 12
Gupta, P., Khare, V., Srivastava, A., Agarwal, J., Mittal, V., Sonkar, V., Saxena, S., Agarwal, A., & Jain, A. (2023) · Journal of Investigative Medicine · doi:10.1177/10815589231171402 · Cited by 11
Accuracy of Artificial Intelligence-Designed Dental Crowns: A Scoping Review of In-Vitro Studies
Kong, H.-J., & Kim, Y. (2025) · Applied Sciences · doi:10.3390/app15189866 · Cited by 11
Guo, K., Yan, H., Kui, X., Sehdev, P., Chi, T., Zhang, R., & Li, J. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.10.029 · Cited by 11
Artificial Intelligence for Global Healthcare
Parsa, A. D., Hakkim, S., Vinnakota, D., Mahmud, I., Bulsari, S., Dehghani, L., Pulikkottil, A. T., Sivasubramanian, M., & Kabir, R. (2023) · Medical virology · doi:10.1007/978-981-99-0369-6_1 · Cited by 11
Ha, E.-G., Jeon, K. J., Lee, C., Kim, H.-S., & Han, S. (2023) · Dentomaxillofacial Radiology · doi:10.1259/dmfr.20220413 · Cited by 11
Musculoskeletal trauma imaging in the era of novel molecular methods and artificial intelligence
Klontzas, M. E., Papadakis, G. Z., Marias, K., & Karantanas, A. H. (2020) · Injury · doi:10.1016/j.injury.2020.09.019 · Cited by 11
Kim, K., Cho, K., Kim, M., Lee, K. H., Lee, S. J., Lee, S. M., Lee, K. H., & Kim, N. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106705 · Cited by 11
Metadata
- Authors
- Park, S. H., & Han, K.
- Affiliations
- Yonsei University (KR)
- Venue
- Radiology
- Published
- 2018-03
- Volume
- 286
- Issue
- 3
- Pages
- 800-809
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2783687327
- Cited by
- 894
- References
- 49
- Citations by year
- 2026: 123 · 2025: 98 · 2024: 96 · 2023: 135 · 2022: 111 · 2021: 121 · 2020: 103 · 2019: 80 · 2018: 25
- FWCI
- 1.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Ministry of Trade, Industry and Energy
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- artificial intelligence in healthcare, deep neural network, medical image diagnosis, prediction model, discrimination and calibration, overfitting, spectrum bias, external validation, disease prevalence, outcomes research, high-dimensional models
- MeSH terms
- Artificial Intelligence, Data Interpretation, Statistical, Diagnostic Imaging, Humans, Image Interpretation, Computer-Assisted, Terminology as Topic, Predictive Value of Tests, Technology Assessment, Biomedical, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.