Methodologic Guide for Evaluating Clinical Performance and Effect of Artificial Intelligence Technology for Medical Diagnosis and Prediction
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
The use of artificial intelligence in medicine is currently an issue of great interest, especially with regard to the diagnostic or predictive analysis of medical images. Adoption of an artificial intelligence tool in clinical practice requires careful confirmation of its clinical utility. Herein, the authors explain key methodology points involved in a clinical evaluation of artificial intelligence technology for use in medicine, especially high-dimensional or overparameterized diagnostic or predictive models in which artificial deep neural networks are used, mainly from the standpoints of clinical epidemiology and biostatistics. First, statistical methods for assessing the discrimination and calibration performances of a diagnostic or predictive model are summarized. Next, the effects of disease manifestation spectrum and disease prevalence on the performance results are explained, followed by a discussion of the difference between evaluating the performance with use of internal and external datasets, the importance of using an adequate external dataset obtained from a well-defined clinical cohort to avoid overestimating the clinical performance as a result of overfitting in high-dimensional or overparameterized classification model and spectrum bias, and the essentials for achieving a more robust clinical evaluation. Finally, the authors review the role of clinical trials and observational outcome studies for ultimate clinical verification of diagnostic or predictive artificial intelligence tools through patient outcomes, beyond performance metrics, and how to design such studies. © RSNA, 2018
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This common scenario means that estimations of AI performance based on internal test data may overestimate its real-world performance on external data [73–75].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [74]
In the citing paper
“Similar to internal validation of the AIPM, external AIPM model validation can be based on discrimination (area under the receiver operating characteristic curve, sensitivity, specificity, positive and negative predictive values), calibration (calibration plot)5,6,13,17,49,56,78, and Decision Curve Analysis5,6,17,49,68,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“To ensure the generalizability of the AIPM to the intended healthcare setting, developers are advised to extensively validate the model for representative data from that setting6–8,10,13–15,24,27,56,64,66,68,72,77,79,87,88 (see phase 1, Representativeness).”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“The literature recommends to assess generalizability on external data from a different time period, place, or healthcare setting3,7,8,10,16,17,27,56,68,72,79.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“Although prospective validation is rare, it is preferred by the literature5,14,56, as it provides a better idea of the AIPM’s true applicability to medical practice and allows the healthcare professionals to identify and review errors in real time19,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
(2016) · Cited by 11,679
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293
Lakhani, P., & Sundaram, B. (2017) · Radiology · doi:10.1148/radiol.2017162326 · Cited by 1,775
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Radiology · doi:10.1148/radiol.2015151516 · Cited by 1,337
Deep Learning in Medical Imaging: General Overview
Lee, J.-G., Jun, S., Cho, Y.-W., Lee, H., Kim, G. B., Seo, J. B., & Kim, N. (2017) · Korean Journal of Radiology · doi:10.3348/kjr.2017.18.4.570 · Cited by 1,276
Research works printed but not linked: 3
- Steyerberg EW, 2010, Clinical prediction models: a practical approach to development, validation, and updating, 83
- Steyerberg EW, 2010, Clinical prediction models: a practical approach to development, validation, and updating, 299
- Newman TB, 2013, Designing clinical research, 117
Cited by
Items 301–400 of 1,000
Adili, D., Mohetaer, A., & Zhang, W. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.111249 · Cited by 30
Bartoli, A., Fournel, J., Bentatou, Z., Habib, G., Lalande, A., Bernard, M., Boussel, L., Pontana, F., Dacher, J., Ghattas, B., et al. (2020) · Radiology Artificial Intelligence · doi:10.1148/ryai.2020200021 · Cited by 30
Thrall, J. H., Fessell, D. P., & Pandharipande, P. V. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.07.010 · Cited by 30
Wong, L. M., Ai, Q. Y. H., Mo, F. K. F., Poon, D. M. C., & King, A. D. (2021) · Japanese Journal of Radiology · doi:10.1007/s11604-021-01092-x · Cited by 30
Mahmood, U. A., Shrestha, R., Bates, . D. B., Mannelli, L., Corrias, G., Erdi, Y. E., & Kanan, C. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.671015 · Cited by 30
“The standard approach used to identify system failures involves testing with held-out development or generalization test datasets (28).”
MethodAutomated detection and analysis of subdural hematomas using a machine learning algorithm
Colasurdo, M., Leibushor, N., Robledo, A., Vasandani, V., Luna, Z. A. E., Rao, A. S., Garcia, R., Srinivasan, V. M., Sheth, S. A., Avni, N., et al. (2022) · Journal of neurosurgery · doi:10.3171/2022.8.jns22888 · Cited by 30
Artificial Intelligence‐Based Detection and Numbering of Dental Implants on Panoramic Radiographs
Balel, Y., Sağtaş, K., Teke, F., & Kurt, M. A. (2025) · Clinical Implant Dentistry and Related Research · doi:10.1111/cid.70000 · Cited by 29
Lea, W. W.-I., Hong, S., Nam, H., Kang, W., Yang, Z., & Noh, E.-J. (2022) · Scientific Reports · doi:10.1038/s41598-022-05282-z · Cited by 29
“External validation is important in verifying an overparameterized diagnostic or predictive classification model that is built with high-dimensional data such as deep learning algorithms to analyze radiologic images 18 .”
BackgroundDebnath, S. (2023) · World Journal of Advanced Research and Reviews · doi:10.30574/wjarr.2023.19.1.1335 · Cited by 29
“One example of the use of AI in healthcare is the development of computer-aided diagnosis (CAD) systems, which use machine learning algorithms to analyze medical images and assist radiologists in identifying abnormalities [15].”
MethodAlmarashdeh, I., Alsmadi, M. K., Jaradat, G. M., Althunibat, A., Albahussain, S. A., Qawqzeh, Y., Badawi, U. A., & Farag, T. (2018) · Journal of Computer Science · doi:10.3844/jcssp.2018.453.465 · Cited by 28
Development of a Deep Learning–Based Model for Diagnosing Breast Nodules With Ultrasound
Li, J., Bu, Y., Lu, S., Pang, H., Luo, C., Liu, Y., & Qian, L. (2020) · Journal of Ultrasound in Medicine · doi:10.1002/jum.15427 · Cited by 28
“There have now been many clinical trials of AI, which have shown that it can achieve an accurate analysis and diagnosis of medical images.(9,10) We therefore aimed to assess the value of deep learning (DL) for improving the diagnosis of breast nodules.”
BackgroundZhang, J., Liu, Y., Mitsuhashi, T., & Matsuo, T. (2021) · Journal of Ophthalmology · doi:10.1155/2021/8883946 · Cited by 28
“(3) ,e performance of the DL model is affected by the image quality and disease manifestation [37, 39].”
InfluentialAdversarial training for prostate cancer classification using magnetic resonance imaging
Hu, L., Zhou, D., Guo, X., Xu, W., Wei, L., & Zhao, J. (2022) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-21-1089 · Cited by 28
Hwang, E. J., Goo, J. M., Nam, J. G., Park, C. M., Hong, K. J., & Kim, K. H. (2023) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0651 · Cited by 28
“Moreover, improving the accuracy of CR interpretation does not necessarily result in improvements in patient management, patient outcomes, or workflow efficiency [23].”
Ruitenbeek, H., Oei, E. H. G., Visser, J. J., & Kijowski, R. (2024) · Skeletal Radiology · doi:10.1007/s00256-024-04684-6 · Cited by 28
“It is generally recommended to perform multi-institutional studies with consecutive cases and use external validation to enhance the generalizability and clinical impact of the DL models [7, 44–46].”
Wei, B. (2024) · JMIR Medical Education · doi:10.2196/64284 · Cited by 28
Kim, D. W., Jang, H. Y., Ko, Y., Son, J. H., Kim, P. H., Kim, S., Lim, J. S., & Park, S. H. (2020) · PLoS ONE · doi:10.1371/journal.pone.0238908 · Cited by 28
“Prior to participation in the current study, the reviewers received reading materials [12, 15, 17, 18] and had an offline discussion to review their contents.”
BackgroundSelf-citationClinical Artificial Intelligence Applications in Radiology
Lee, S., & Summers, R. M. (2021) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2021.07.001 · Cited by 28
Artificial Intelligence in Medical Imaging: From Theory to Clinical Practice
Morra, L., Delsanto, S., & Correale, L. (2019) · CRC Press eBooks · doi:10.1201/9780367229184 · Cited by 28
Chalkidou, A., Shokraneh, F., Kijauskaite, G., Taylor‐Phillips, S., Halligan, S., Wilkinson, L., Glocker, B., Garrett, P., Denniston, A. K., Mackie, A., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00186-8 · Cited by 27
“12,19 Broadly, temporal independency refers to the use of data collected in different time periods but from the same institutions that provided development data.”
InfluentialPham, N., Hill, V. B., Rauschecker, A. M., Lui, Y. W., Niogi, S. N., Filippi, C. G., Chang, P., Zaharchuk, G., & Wintermark, M. (2023) · American Journal of Neuroradiology · doi:10.3174/ajnr.a7850 · Cited by 27
“31 Scientific evidence that supports technical efficacy is often in the form of retrospective studies and includes peer-reviewed information about the AI model ’ s data quality, development, and performance metrics, benchmarked against similar or alternatively accepted methods in the literature.”
BackgroundMethodMRI-Based Radiomics in Bladder Cancer: A Systematic Review and Radiomics Quality Score Assessment
Boca, B., Caraiani, C., Telecan, T., Pintican, R. M., Lebovici, A., Andraș, I., Crişan, N., Pavel, A., Dioşan, L. S., Bálint, Z., et al. (2023) · Diagnostics · doi:10.3390/diagnostics13132300 · Cited by 27
Czakó, L., Šufliarsky, B., Šimko, K., Soviš, M., Vidová, I., Farska, J., Lifková, M., Hamar, T., & Gális, B. (2024) · Bioengineering · doi:10.3390/bioengineering11070679 · Cited by 27
“The surgeon’s movements are translated into precise movements of the robotic arms, allowing for enhanced dexterity, stability, and maneuverability in head and neck surgery [56,57].”
BackgroundAhluwalia, M., Abdalla, M., Sanayei, J., Seyyed-Kalantari, L., Hussain, M., Ali, A., & Fine, B. (2023) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220270 · Cited by 26
Medical Treatment of Acromegaly: Navigating the Present, Shaping the Future
Gadelha, M. R., Wildemberg, L. E., Marques, N. V., & Kasuki, L. (2025) · Endocrine Reviews · doi:10.1210/endrev/bnaf020 · Cited by 26
Kim, C., Lee, G., Oh, H., Jeong, G.-J., Kim, S. W., Chun, E. J., Kim, Y., Lee, J., & Yang, D. H. (2021) · European Radiology · doi:10.1007/s00330-021-08296-9 · Cited by 26
Artificial Intelligence in Breast Imaging: Challenges of Integration Into Clinical Practice
Ozcan, B. B., Patel, B., Banerjee, I., & Doğan, B. E. (2023) · Journal of Breast Imaging · doi:10.1093/jbi/wbad007 · Cited by 26
Gonzalez, R., Nejat, P., Saha, A., Campbell, C. J. V., Norgan, A. P., & Lokker, C. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100348 · Cited by 27
Current status of deep learning applications in abdominal ultrasonography
Song, K. D. (2020) · ULTRASONOGRAPHY · doi:10.14366/usg.20085 · Cited by 26
“In particular, it is desirable to perform external validation of a model’s performance in a clinical cohort that represents the target population of the developed model using prospectively collected data [12].”
BackgroundMethodKang, N. G., Suh, Y. J., Han, K., Kim, Y. J., & Choi, B. W. (2020) · Korean Journal of Radiology · doi:10.3348/kjr.2020.0099 · Cited by 26
“The use of the term “validation” was similar to that used in the medicine- and health-related literature, indicating a process used to verify model performance (16).”
Self-citationRadiomics and deep learning in liver diseases
Sung, Y. S., Park, B., Park, H. J., & Lee, S. S. (2021) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.15414 · Cited by 26
Mulliqi, N., Blilie, A., Ji, X., Szolnoky, K., Olsson, H., Boman, S. E., Titus, M., Gonzalez, G. M., Mielcarz, J. A., Valkonen, M., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-78996-7 · Cited by 25
Subtle anomaly detection: Application to brain MRI analysis of de novo Parkinsonian patients
Muñoz-Ramírez, V., Kmetzsch, V., Forbes, F., Meoni, S., Moro, E., & Dojat, M. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102251 · Cited by 25
“, 2019) and few are evaluated in clinical routine conditions (Park and Han, 2018).”
Lee, J. H., Hwang, E. J., Kim, H., & Park, C. M. (2022) · Translational Lung Cancer Research · doi:10.21037/tlcr-21-1012 · Cited by 25
“Therefore, the internal test may substantially overestimate the performance of a DL algorithm (15,16).”
InfluentialArtificial Intelligence in Lung Imaging
Choe, J., Lee, S. M., Hwang, H. J., Yun, J., Kim, N., & Seo, J. B. (2022) · Seminars in Respiratory and Critical Care Medicine · doi:10.1055/s-0042-1755571 · Cited by 26
Saleh, M., AlHamaydeh, M. H., & Zakaria, M. (2023) · Engineering Structures · doi:10.1016/j.engstruct.2023.115675 · Cited by 25
Xu, Y., Zheng, S. J., Tian, Q., Kou, Z., Li, W., Xie, X., & Wu, X. (2024) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.29403 · Cited by 25
Theories of Change in Reality: Strengths, Limitations and Future Directions
Koleros, A., Adrien, M.-H., & Tyrrell, T. (2024) · Routledge eBooks · doi:10.4324/9781032669618 · Cited by 25
Zhang, J., Li, X., Li, Y., Wang, M., Huang, B., Yao, S., & Shen, L. (2019) · Brain Imaging and Behavior · doi:10.1007/s11682-019-00186-5 · Cited by 25
Swillens, J. E. M., Nagtegaal, I. D., Engels, S., Lugli, A., Hermens, R., & van der Laak, J. A. W. M. (2023) · Oncogene · doi:10.1038/s41388-023-02797-1 · Cited by 24
Ahmed, K. A., Humaira, I., Khan, A. R., Hasan, M. S., Islam, M., Roy, A., Karim, M., Uddin, M., Mohammad, A., & Xames, M. D. (2025) · PLoS ONE · doi:10.1371/journal.pone.0326221 · Cited by 24
“Efficiently and accurately diagnosing breast cancer is a major challenge in both the bioinformatics field and the medical science field [2].”
Klontzas, M. E., Stathis, I., Spanakis, K., Zibis, A. H., Marias, K., & Karantanas, A. H. (2022) · Diagnostics · doi:10.3390/diagnostics12081870 · Cited by 24
“Curation of the dataset prior to model development and training included splitting into training and validation cohorts, with a ratio of 70:30, and images of each group were processed using grey level harmonization and histogram normalization to reduce MR scanner-related variability across our dataset while avoiding d…”
MethodLehnen, N. C., Haase, R., Schmeel, F. C., Vatter, H., Dorn, F., Radbruch, A., & Paech, D. (2022) · American Journal of Neuroradiology · doi:10.3174/ajnr.a7695 · Cited by 24
Uddin, K. M. M., Biswas, N., Rikta, S. T., Dey, S. K., & Qazi, A. (2023) · Engineering Reports · doi:10.1002/eng2.12666 · Cited by 24
“47 Another justification is that the implementation is simple to use, well-documented, and has active community support.”
BackgroundArtificial intelligence and radiomics in magnetic resonance imaging of rectal cancer: a review
Costanzo, G., Ascione, R., Ponsiglione, A., Tucci, A. G., Dell’Aversana, S., Iasiello, F., & Cavaglià, E. (2023) · Exploration of Targeted Anti-tumor Therapy · doi:10.37349/etat.2023.00142 · Cited by 24
Artificial Intelligence and Evaluation: Emerging Technologies and Their Implications for Evaluation
Nielsen, S. B., Rinaldi, F. M., & Petersson, G. J. (2024) · Routledge eBooks · doi:10.4324/9781003512493 · Cited by 25
The Importance of Imaging Informatics and Informaticists in the Implementation of AI
Cook, T. S. (2019) · Academic Radiology · doi:10.1016/j.acra.2019.10.002 · Cited by 24
Examination of the diaphragm in obstructive sleep apnea using ultrasound imaging
Molnár, V., Molnár, A., Lakner, Z., Tárnoki, D. L., Tárnoki, Á. D., Jokkel, Z., Szabó, H., Dienes, A., Angyal, E., Németh, F., et al. (2021) · Sleep And Breathing · doi:10.1007/s11325-021-02472-3 · Cited by 23
van de Sande, D., van Genderen, M. E., Verhoef, C. K., Huiskens, J., Gommers, D. A. M. P. J., van Unen, E., Schasfoort, R. A., Schepers, J., van Bommel, J., & Grünhagen, D. J. (2022) · Surgery · doi:10.1016/j.surg.2022.03.031 · Cited by 23
Hesse, R., Raal, F. J., Blom, D. J., & George, J. A. (2022) · Circulation Genomic and Precision Medicine · doi:10.1161/circgen.121.003324 · Cited by 23
Schwartzmann, B., Dhami, P., Uher, R., Lam, R. W. M., Frey, B. N., Milev, R. V., Müller, D. J., Blier, P., Soares, C. N., Parikh, S. V., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.36094 · Cited by 23
Next-Generation Sequencing for the Detection of Microbial Agents in Avian Clinical Samples
Afonso, C. L., & Afonso, A. M. (2023) · Veterinary Sciences · doi:10.3390/vetsci10120690 · Cited by 23
Application of deep learning in the diagnosis and evaluation of ulcerative colitis disease severity
Jiang, X., Luo, X., Nan, Q., Ye, Y., Miao, Y., & Miao, J. (2023) · Therapeutic Advances in Gastroenterology · doi:10.1177/17562848231215579 · Cited by 23
“14 Studies have shown that deep learning models trained on specific medical images can achieve expert-level evaluations.”
Christou, C. D., & Tsoulfas, G. (2023) · World Journal of Gastroenterology · doi:10.3748/wjg.v29.i48.6168 · Cited by 23
“On the other hand, overfitting can be described as the inability of the model to retain its performance in datasets other than the training dataset[47].”
Prediction and Diagnosis of Breast Cancer Using Machine and Modern Deep Learning Models
Devi, S., Kaul-Ghanekar, R., Pande, J., Dumbre, D., Chavan, R. G., & Gupta, H. (2024) · Asian Pacific Journal of Cancer Prevention · doi:10.31557/apjcp.2024.25.3.1077 · Cited by 23
“ML algorithms have shown promise in predicting survival and determining the malignancy of breast tumors in the early stages [6].”
BackgroundBouzid, A. I. H., Bui, S., Benlala, I., Berger, P., Hutt, A., Liberge, R., Habert, P., Gaubert, J., Baqué-Juston, M., Morel, B., et al. (2024) · European Radiology · doi:10.1007/s00330-024-11019-5 · Cited by 23
Ma, S., Zhang, M., Sun, W., Gao, Y., Jing, M., Gao, L., & Wu, Z. (2025) · Healthcare and Rehabilitation · doi:10.1016/j.hcr.2024.100006 · Cited by 23
Hwang, E.-J., Jung, J., Lee, S. K., Lee, S., & Jee, W.-H. (2019) · Scientific Reports · doi:10.1038/s41598-019-42579-y · Cited by 23
“This sampling pattern carries a risk of biases that the samples might not adequately reflect the spectrum in real clinical practice 27 .”
BackgroundMethodLazăr, D. C., Avram, M. F., Faur, A. C., Goldiș, A., Romosan, I., Tăban, S. M., & Cornianu, M. (2020) · Medicina · doi:10.3390/medicina56070364 · Cited by 24
Humans and machines: Moving towards a more symbiotic approach to learning clinical reasoning
Pinnock, R., McDonald, J., Ritchie, D., & Durning, S. J. (2019) · Medical Teacher · doi:10.1080/0142159x.2019.1679361 · Cited by 23
Placenta Accreta Spectrum Disorders and Radiomics: Systematic review and quality appraisal
Stanzione, A., Verde, F., Cuocolo, R., Romeo, V., Paolo Mainenti, P., Brunetti, A., & Maurea, S. (2022) · European Journal of Radiology · doi:10.1016/j.ejrad.2022.110497 · Cited by 22
Guerrini Usubini, A., Varallo, G., Giusti, E. M., Cattivelli, R., Granese, V., Consoli, S., Bastoni, I., Volpi, C., & Castelnuovo, G. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.861341 · Cited by 22
Bolocan, V.-O., Mitoi, A., Nicu-Canareica, O., Băean, M.-L., Medar, C., & Popa, G.-A. (2025) · Journal of Imaging · doi:10.3390/jimaging11100335 · Cited by 22
“However, this raises important questions about fair comparison with radiologists who also have access to clinical information, and whether reported AI superiority reflects true algorithmic advantage or simply more comprehensive data integration [34].”
Wu, X., & Dai, W. (2025) · Cancers · doi:10.3390/cancers17213408 · Cited by 22
Christou, C. D., & Tsoulfas, G. (2022) · World Journal of Gastrointestinal Oncology · doi:10.4251/wjgo.v14.i4.765 · Cited by 22
Gunzer, F., Jantscher, M., Hassler, E. M., Kau, T., & Reishofer, G. (2022) · Insights into Imaging · doi:10.1186/s13244-022-01311-7 · Cited by 22
Colasurdo, M., Shalev, D., Robledo, A., Vasandani, V., Luna, Z. A. E., Rao, A. S., Garcia, R., Edhayan, G., Srinivasan, V. M., Sheth, S. A., et al. (2023) · Journal of neurosurgery · doi:10.3171/2023.1.jns222304 · Cited by 22
Yearley, A. G., Goedmakers, C. M. W., Panahi, A., Doucette, J. A., Rana, A., Ranganathan, K., & Smith, T. R. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102607 · Cited by 22
Shah, A. K., Lavu, M. S., Hecht, C. J., Burkhart, R. J., & Kamath, A. F. (2023) · Arthroplasty · doi:10.1186/s42836-023-00209-z · Cited by 22
“To improve the reproducibility of the AI algorithms, future studies ought to conduct tests against external datasets [38, 42, 43, 47, 49, 50].”
BackgroundMethodChan, P. Z., Ramli, M. A. I. B., & Chew, H. S. J. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107636 · Cited by 22
Challenges of Machine Learning and AI (What Is Next?), Responsible and Ethical AI
Gkontra, P., Quaglio, G., Garmendia, A. T., & Lekadir, K. (2023) · Springer eBooks · doi:10.1007/978-3-031-36678-9_17 · Cited by 22
Singh, V., Sarkar, S., Gaur, V., Grover, S., & Singh, O. P. (2024) · Indian Journal of Psychiatry · doi:10.4103/indianjpsychiatry.indianjpsychiatry_926_23 · Cited by 22
“[15] ETHICAL ISSUES However, the use of AI in mental health is associated with significant ethical challenges [Table 4].”
Mäenpää, S. M., & Korja, M. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105523 · Cited by 22
Antolín, A., Rosón, N., Mast, R., Arce, J., Almodovar, R., Cortada, R., Maceda, A., Escobar, M., Trilla, E., & Moróte, J. (2024) · Cancers · doi:10.3390/cancers16172951 · Cited by 22
Liang, X., Fang, J., Li, H., Yang, X., Ni, D., Zeng, F., & Chen, Z. (2020) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2020.07.020 · Cited by 22
Cha, D. I., Kang, T. W., Min, J. H., Joo, I., Sinn, D. H., Ha, S. Y., Kim, K., Lee, G., & Yi, J. (2021) · ULTRASONOGRAPHY · doi:10.14366/usg.20179 · Cited by 22
Multivoxel pattern analysis of structural MRI in children and adolescents with conduct disorder
Zhang, J., Cao, W., Wang, M., Wang, N., Yao, S., & Huang, B. (2018) · Brain Imaging and Behavior · doi:10.1007/s11682-018-9953-6 · Cited by 22
Predicting the prevalence of lung cancer using feature transformation techniques
Munawar, Z., Ahmad, F. B., Alanazi, S., Nisar, K. S., Khalid, M., Anwar, M., & Murtaza, K. (2022) · Egyptian Informatics Journal · doi:10.1016/j.eij.2022.08.002 · Cited by 21
Liu, Y., Feng, Y., Qian, L., Wang, Z., & Hu, X. (2023) · Experimental Biology and Medicine · doi:10.1177/15353702231220664 · Cited by 21
“20,26 Both AI and radiologists make similar mistakes when differentiating benign and malignant thyroid neoplasms.”
BackgroundRemote Assessment of Eczema Severity via AI-powered Skin Image Analytics: A Systematic Review
Huang, L., Tang, W. H., Attar, R., Gore, C., Williams, H. C., Čustović, A., & Tanaka, R. J. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102968 · Cited by 21
Apprehensions about Excessive Belief in Digital Therapeutics: Points of Concern Excluding Merits
Kim, H. (2020) · Journal of Korean Medical Science · doi:10.3346/jkms.2020.35.e373 · Cited by 21
Proposals for the use of artificial intelligence in emergency radiology
Jacques, T., Fournier, L. S., Zins, M., Adamsbaum, C., Chaumoître, K., Feydy, A., Millet, I., Montaudon, M., Bérégi, J.-P., Bartoli, J., et al. (2020) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2020.11.003 · Cited by 21
Durant, T. J. S., Dudgeon, S. N., McPadden, J., Simpson, A., Price, N. D., Schulz, W. L., Torres, R., & Olson, E. M. (2021) · Clinical Chemistry · doi:10.1093/clinchem/hvab237 · Cited by 21
How to Clearly and Accurately Report Odds Ratio and Hazard Ratio in Diagnostic Research Studies?
Park, S. H., & Han, K. (2022) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0249 · Cited by 21
“In retrospective diagnostic research studies using convenience sampling, even though the study samples are not precisely collected in a case-control manner, they may be similar to case-control samples because of the various participant selection steps involved [4].”
BackgroundSelf-citationZhang, Y., Liu, M., Zhang, L., Wang, L., Zhao, K., Hu, S., Chen, X., & Xie, X. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.55113 · Cited by 20
Hao, X., Wang, Y., Li, K., Zhu, T., & Herasevich, V. (2025) · Anesthesiology and Perioperative Science · doi:10.1007/s44254-025-00136-z · Cited by 20
“To develop models with favorable performance and generalizability, a multicenter, prospective data is preferred over a single-center retrospective cohort dataset for higher representativeness and quality of data, more closely matches the real-life conditions [20, 59].”
Waldock, W. J., Guni, A., Darzi, A. W., & Ashrafian, H. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001310 · Cited by 22
“Terms like AUC and AUROC were used interchangeably, often without clear definitions [16].”
Xue, H., Wang, S.-Y., Cui, L., & Hong, K. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.18.20436 · Cited by 20
Point of Sale (POS) Network with Embedded Fingerprint Biometric Authentication
Alzame, H. A., Alshabanah, M., & Alsmadi, M. K. (2019) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst119659 · Cited by 20
“Today, the use of Artificial Intelligence (AI) algorithms is expansive, particularly in providing solutions to challenging problems including patterns recognition and retrieval of information [26, 39, 43, 47-61], image segmentation [11, 12, 22, 42, 62-66], analysis of medical images [67-71], Learning Management System…”
MethodArtificial Intelligence in Diagnosis of Oral Diseases: A Systematic Review
Kolarkodi, S. H., & Alotaibi, K. Z. (2023) · The Journal of Contemporary Dental Practice · doi:10.5005/jp-journals-10024-3465 · Cited by 20
Supervised machine learning for microbiomics: Bridging the gap between current and best practices
Dudek, N. K., Chakhvadze, M., Kobakhidze, S., Kantidze, O. L., & Gankin, Y. V. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100607 · Cited by 20
Principles for evaluating the clinical implementation of novel digital healthcare devices
Park, S. H., Do, K.-H., Choi, J.-I., Sim, J. S., Yang, D. M., Eo, H., Woo, H., Lee, J. M., Jung, S. E., & Oh, J. H. (2018) · Journal of the Korean Medical Association · doi:10.5124/jkma.2018.61.12.765 · Cited by 18
Self-citationSemNet: Using Local Features to Navigate the Biomedical Concept Graph
Sedler, A. R., & Mitchell, C. S. (2019) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2019.00156 · Cited by 20
“, 2012), clinical medicine decision-making (Park and Han, 2018), as well as narrowly defined experimental examination of multi-factorial diseases (Foley et al.”
BackgroundLopes, F. P. P. L., Kitamura, F., Prado, G. F., Kuriki, P. E. D. A., Garcia, M. R. T., & COVID-AI-Brasil (2021) · PLoS ONE · doi:10.1371/journal.pone.0245384 · Cited by 20
Osapoetra, L. O., Dasgupta, A., DiCenzo, D., Fatima, K., Quiaoit, K., Saifuddin, M., Karam, I., Poon, I., Husain, Z. A., Tran, W. T., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-85221-6 · Cited by 20
Kim, G. R., Lee, E., Kim, H. R., Yoon, J. H., Park, V. Y., & Kwak, J. Y. (2021) · American Journal of Neuroradiology · doi:10.3174/ajnr.a7149 · Cited by 20
“5,6,11,29 While artificial intelligence (AI) is not yet considered ready for a clinical setting, 31 computer software is already thought to have several strong advantages over radiologists because its use can overcome human variation and provide diagnostic reproducibility and consistency in image…”
BackgroundRegulatory Approval versus Clinical Validation of Artificial Intelligence Diagnostic Tools
Park, S. H. (2018) · Radiology · doi:10.1148/radiol.2018181310 · Cited by 20
Self-citationA research roadmap for AI opportunities in student assessment for medical education
Rezaei-Zadeh, M., & Cerbin‐Koczorowska, M. (2025) · BMC Medical Education · doi:10.1186/s12909-025-08078-7 · Cited by 19
Parrales–Bravo, F., Barrio, A. A. D., Gago‐Veiga, A. B., Gallego, M. M., Piñero, M. R., Peral, Á. L. G., Džeroski, S., & Ayala, J. L. (2019) · IEEE Access · doi:10.1109/access.2019.2927429 · Cited by 19
Early detection of suspicious lymph nodes in differentiated thyroid cancer
Frasca, F., Piticchio, T., Moli, R. L., Tumino, D., Cannavò, S., Ruggeri, R. M., Campennì, A., & Giovanella, L. (2022) · Expert Review of Endocrinology & Metabolism · doi:10.1080/17446651.2022.2112176 · Cited by 19
“Although several studies indicated the applicability of these tools in support of diagnostic imaging [40,41], only few of them are beginning to verify their”
Background
Metadata
- Authors
- Park, S. H., & Han, K.
- Affiliations
- Yonsei University (KR)
- Venue
- Radiology
- Published
- 2018-03
- Volume
- 286
- Issue
- 3
- Pages
- 800-809
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2783687327
- Cited by
- 894
- References
- 49
- Citations by year
- 2026: 123 · 2025: 98 · 2024: 96 · 2023: 135 · 2022: 111 · 2021: 121 · 2020: 103 · 2019: 80 · 2018: 25
- FWCI
- 1.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Ministry of Trade, Industry and Energy
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- artificial intelligence in healthcare, deep neural network, medical image diagnosis, prediction model, discrimination and calibration, overfitting, spectrum bias, external validation, disease prevalence, outcomes research, high-dimensional models
- MeSH terms
- Artificial Intelligence, Data Interpretation, Statistical, Diagnostic Imaging, Humans, Image Interpretation, Computer-Assisted, Terminology as Topic, Predictive Value of Tests, Technology Assessment, Biomedical, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.