Methodologic Guide for Evaluating Clinical Performance and Effect of Artificial Intelligence Technology for Medical Diagnosis and Prediction
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
The use of artificial intelligence in medicine is currently an issue of great interest, especially with regard to the diagnostic or predictive analysis of medical images. Adoption of an artificial intelligence tool in clinical practice requires careful confirmation of its clinical utility. Herein, the authors explain key methodology points involved in a clinical evaluation of artificial intelligence technology for use in medicine, especially high-dimensional or overparameterized diagnostic or predictive models in which artificial deep neural networks are used, mainly from the standpoints of clinical epidemiology and biostatistics. First, statistical methods for assessing the discrimination and calibration performances of a diagnostic or predictive model are summarized. Next, the effects of disease manifestation spectrum and disease prevalence on the performance results are explained, followed by a discussion of the difference between evaluating the performance with use of internal and external datasets, the importance of using an adequate external dataset obtained from a well-defined clinical cohort to avoid overestimating the clinical performance as a result of overfitting in high-dimensional or overparameterized classification model and spectrum bias, and the essentials for achieving a more robust clinical evaluation. Finally, the authors review the role of clinical trials and observational outcome studies for ultimate clinical verification of diagnostic or predictive artificial intelligence tools through patient outcomes, beyond performance metrics, and how to design such studies. © RSNA, 2018
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This common scenario means that estimations of AI performance based on internal test data may overestimate its real-world performance on external data [73–75].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [74]
In the citing paper
“Similar to internal validation of the AIPM, external AIPM model validation can be based on discrimination (area under the receiver operating characteristic curve, sensitivity, specificity, positive and negative predictive values), calibration (calibration plot)5,6,13,17,49,56,78, and Decision Curve Analysis5,6,17,49,68,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“To ensure the generalizability of the AIPM to the intended healthcare setting, developers are advised to extensively validate the model for representative data from that setting6–8,10,13–15,24,27,56,64,66,68,72,77,79,87,88 (see phase 1, Representativeness).”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“The literature recommends to assess generalizability on external data from a different time period, place, or healthcare setting3,7,8,10,16,17,27,56,68,72,79.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“Although prospective validation is rare, it is preferred by the literature5,14,56, as it provides a better idea of the AIPM’s true applicability to medical practice and allows the healthcare professionals to identify and review errors in real time19,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
(2016) · Cited by 11,679
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293
Lakhani, P., & Sundaram, B. (2017) · Radiology · doi:10.1148/radiol.2017162326 · Cited by 1,775
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Radiology · doi:10.1148/radiol.2015151516 · Cited by 1,337
Deep Learning in Medical Imaging: General Overview
Lee, J.-G., Jun, S., Cho, Y.-W., Lee, H., Kim, G. B., Seo, J. B., & Kim, N. (2017) · Korean Journal of Radiology · doi:10.3348/kjr.2017.18.4.570 · Cited by 1,276
Research works printed but not linked: 3
- Steyerberg EW, 2010, Clinical prediction models: a practical approach to development, validation, and updating, 83
- Steyerberg EW, 2010, Clinical prediction models: a practical approach to development, validation, and updating, 299
- Newman TB, 2013, Designing clinical research, 117
Cited by
Items 201–300 of 1,000
Benhayoun-Sadafiyine, L., & Lang, D. W. (2021) · Human Systems Management · doi:10.3233/hsm-211179 · Cited by 53
Wang, S., Sun, Y., Li, R., Mao, N., Li, Q., Jiang, T., Chen, Q., Duan, S., Xie, H., & Gu, Y. (2021) · European Radiology · doi:10.1007/s00330-021-08134-y · Cited by 51
Zhang, X., Yang, Y., Shen, Y.-W., Zhang, K.-R., Jiang, Z., Ma, L., Ding, C., Wang, B., Meng, Y., & Liu, H. (2022) · European Radiology · doi:10.1007/s00330-022-08956-4 · Cited by 50
Multiparameter MRI Predictors of Long-Term Survival in Glioblastoma Multiforme
Stringfield, O., Arrington, J. A., Johnston, S. K., Rognin, N. G., Peeri, N. C., Balagurunathan, Y., Jackson, P., Clark-Swanson, K. R., Swanson, K. R., Egan, K. M., et al. (2019) · Tomography · doi:10.18383/j.tom.2018.00052 · Cited by 50
“MATERIALS AND METHODS Discovery Cohort In this work, we have used the terms “discovery” (or training) and “validation” as they are understood in the field of machine learning, namely, to refer to the specific steps of training– validation–test in model development (19).”
MethodFuture possibilities for artificial intelligence in the practical management of hypertension
Koshimizu, H., Kojima, R., & Okuno, Y. (2020) · Hypertension Research · doi:10.1038/s41440-020-0498-x · Cited by 50
“In addition, the possibility of AI to be used as a clinical support system [7–9] and its regulation and validation for social implementation [10, 11] have been discussed.”
BackgroundDeep Learning Based HPV Status Prediction for Oropharyngeal Cancer Patients
Lang, D. M., Peeken, J. C., Combs, S. E., Wilkens, J. J., & Bartzsch, S. H. (2021) · Cancers · doi:10.3390/cancers13040786 · Cited by 50
“To ensure generalizability to images from institutions and scanners not seen during training, testing on external data is inevitable in a medical setting [35].”
BackgroundMorley, J., Morton, C. E., Karpathakis, K., Taddeo, M., & Floridi, L. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3826358 · Cited by 50
Hwang, E. J., Goo, J. M., Yoon, S. H., Beck, K. S., Seo, J. B., Choi, B. W., Chung, M. J., Park, C. M., Jin, K. N., & Lee, S. M. (2021) · Korean Journal of Radiology · doi:10.3348/kjr.2021.0544 · Cited by 43
“org performances in the actual practice remains unclear, since retrospectively collected data may not fully reflect the prevalence and diversity of abnormalities in the actual clinical situation [10,11].”
BackgroundGaffney, H. J., & Mirza, K. (2025) · Academic Pathology · doi:10.1016/j.acpath.2025.100166 · Cited by 49
“77,78 When AI applications are controlled by corporate entities outside the healthcare sector, the focus often shifts toward automating high-volume, lower-complexity tasks to reduce labor costs.”
Han, M., Ha, E. J., & Park, J. H. (2020) · American Journal of Neuroradiology · doi:10.3174/ajnr.a6922 · Cited by 49
“Artificial intelligence (AI)-based computer-aided diagnostic (CAD) systems are projected to substantially influence the field of diagnostic imaging.(1,2) In recent years, the application of AI, particularly deep-learning techniques using convolutional neural networks, has shown promising results in radiology.”
BackgroundLuijten, S. P. R., Wolff, L., Duvekot, M. H. C., van Doormaal, P., Moudrous, W., Kerkhoff, H., Nijeholt, G. J. L. À., Bokkers, R. P. H., Yo, L. S. F., Hofmeijer, J., et al. (2021) · Journal of NeuroInterventional Surgery · doi:10.1136/neurintsurg-2021-017842 · Cited by 49
Hong, W., Hwang, E. J., Lee, J. H., Park, J., Goo, J. M., & Park, C. M. (2022) · Radiology · doi:10.1148/radiol.211706 · Cited by 49
Bang, M., Eom, J., An, C., Kim, S., Park, Y. W., Ahn, S. S., Kim, J., Lee, S., & Lee, S. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01586-2 · Cited by 48
Siontis, G. C. M., Sweda, R., Noseworthy, P. A., Friedman, P. A., Siontis, K. C., & Patel, C. J. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100466 · Cited by 48
Expanding the Reach and Grasp of Lung Cancer Screening
Osarogiagbon, R. U., Yang, P., & Sequist, L. V. D. (2023) · American Society of Clinical Oncology Educational Book · doi:10.1200/edbk_389958 · Cited by 49
Lee, J. H., Park, S., Hwang, E. J., Goo, J. M., Lee, W. Y., Lee, S., Kim, H., Andrews, J. R., & Park, C. M. (2021) · European Radiology · doi:10.1007/s00330-020-07219-4 · Cited by 47
Self-citationHwang, E. J., Hong, J. H., Lee, K. H., Kim, J. I., Nam, J. G., Kim, D., Choi, H., Yoo, S., Goo, J. M., & Park, C. M. (2020) · European Radiology · doi:10.1007/s00330-020-06771-3 · Cited by 47
Tanaka, I., Furukawa, T., & Morise, M. (2021) · Cancer Cell International · doi:10.1186/s12935-021-02165-7 · Cited by 47
“The application of AI in medicine is currently of great interest, especially in the diagnostic and predictive assessment of medical images [33].”
BackgroundBrejnebøl, M. W., Hansen, P., Nybing, J. U., Bachmann, R., Ratjen, U., Hansen, I. V., Lenskjold, A., Axelsen, M., Lundemann, M. J., & Boesen, M. (2022) · European Journal of Radiology · doi:10.1016/j.ejrad.2022.110249 · Cited by 47
Possible Bias in Supervised Deep Learning Algorithms for CT Lung Nodule Detection and Classification
Sourlos, N., Wang, J., Nagaraj, Y., van Ooijen, P. M. A., & Vliegenthart, R. (2022) · Cancers · doi:10.3390/cancers14163867 · Cited by 47
“not introduced from cases that the algorithm was not trained on [55].”
BackgroundDeep Learning Aided Neuroimaging and Brain Regulation
Xu, M., Ouyang, Y., & Yuan, Z. (2023) · Sensors · doi:10.3390/s23114993 · Cited by 46
“Medical imaging is a critical component of medical diagnosis and treatment, and AI has significantly improved the accuracy, efficiency, and speed of medical imaging processes [30,31].”
BackgroundHu, W., Yii, F. S. L., Chen, R., Zhang, X., Shang, X., Kiburg, K., Woods, E., Vingrys, A. J., Zhang, L., Zhu, Z., et al. (2023) · Translational Vision Science & Technology · doi:10.1167/tvst.12.7.14 · Cited by 47
“47 This could be addressed in the future by developing and promoting the utilization of standard reporting checklists for these trials, such as Standards for Reporting Diagnostic accuracy studies - Artificial Intelligence (STARD-AI).”
The AI revolution in glaucoma: Bridging challenges with opportunities
Li, F., Wang, D., Yang, Z., Zhang, Y., Jiang, J., Liu, X., Kong, K., Zhou, F., Tham, C. C. Y., Medeiros, F. A., et al. (2024) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2024.101291 · Cited by 46
Artificial Intelligence in Neuroimaging: Clinical Applications
Choi, K. S., & Sunwoo, L. (2022) · Investigative Magnetic Resonance Imaging · doi:10.13104/imri.2022.26.1.1 · Cited by 46
“Recently, Park et al. (68) have suggested a methodological guide for evaluating the clinical performance of AI.”
BackgroundImpact of artificial intelligence on pathologists’ decisions: an experiment
Meyer, J. M., Khademi, A., Têtu, B., Han, W., Nippak, P. M. D., & Remisch, D. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac103 · Cited by 46
Value assessment of artificial intelligence in medical imaging: a scoping review
Fasterholdt, I., Naghavi‐Behzad, M., Rasmussen, B. S., Kjølhede, T., Skjøth, M. M., Hildebrandt, M. G., & Kidholm, K. (2022) · BMC Medical Imaging · doi:10.1186/s12880-022-00918-y · Cited by 45
“Based on titles and abstracts 166 papers were eligible for the full-text assessment, and a total of 79 papers [11, 12, 19, 33– 108”
BackgroundA review of deep learning approaches for multimodal image segmentation of liver cancer
Wu, C., Chen, Q., Wang, H., Guan, Y., Mian, Z., Huang, C. X., Ruan, C., Song, Q., Jiang, H., Pan, J., et al. (2024) · Journal of Applied Clinical Medical Physics · doi:10.1002/acm2.14540 · Cited by 45
Alam, M. A., Sohel, A., Hasan, K. M., & Ahmad, I. (2024) · Strategic Data Management and Innovation · doi:10.71292/sdmi.v1i01.6 · Cited by 45
Engel, C., Ebel, P., & van Giffen, B. (2021) · Lecture notes in information systems and organisation · doi:10.1007/978-3-030-86797-3_12 · Cited by 45
“Furthermore, the experimental character of AI requires a scientific evaluation of results, which requires change in the STS component of task: As AI systems operate probabilistically, this requires to evaluate the final AI solution or to test minimum viable products (MVPs) in a scientific manner that is reproducible,…”
Clinician's guide to trustworthy and responsible artificial intelligence in cardiovascular imaging
Szabo, L., Raisi-Estabragh, Z., Salih, A., McCracken, C., Ruiz Pujadas, E., Gkontra, P., Kiss, M., Maurovich-Horvath, P., Vago, H., Merkely, B., et al. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.1016032 · Cited by 36
“Given the degradation in performance reported for deep-learning algorithms for medical imaging, it is of paramount importance that the test set consists of independent cohorts to allow for external validation, a key requirement for ensuring the trustworthiness of AI systems (19, 20).”
BackgroundMachine learning applications in precision medicine: Overcoming challenges and unlocking potential
Nilius, H., Tsouka, S., Nagler, M., & Masoodi, M. (2024) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2024.117872 · Cited by 44
Artificial intelligence in orthopedic implant model classification: a systematic review
Ren, M., & Yi, P. H. (2021) · Skeletal Radiology · doi:10.1007/s00256-021-03884-8 · Cited by 44
Suh, Y. J., Kim, C., Lee, J., Oh, H., Kang, H., Kim, Y., & Yang, D. H. (2022) · European Radiology · doi:10.1007/s00330-022-09117-3 · Cited by 43
“The performance of DL algorithms can rapidly degrade when only slight variations exist in the input data [25].”
BackgroundMotahari, A., Barr, R. G., Han, M. K., Anderson, W. H., Barjaktarević, I., Bleecker, E. R., Comellas, A. P., Cooper, C. B., Couper, D. J., Hansel, N. N., et al. (2023) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.202209-1698pp · Cited by 43
Biodegradation of ciprofloxacin using machine learning tools: Kinetics and modelling
Kamal, N., Saha, A. K., Singh, E., Pandey, A., & Bhargava, P. C. (2024) · Journal of Hazardous Materials · doi:10.1016/j.jhazmat.2024.134076 · Cited by 43
Promises of artificial intelligence in neuroradiology: a systematic technographic review
Olthof, A. W., van Ooijen, P. M. A., & Rezazade Mehrizi, M. H. (2020) · Neuroradiology · doi:10.1007/s00234-020-02424-w · Cited by 36
“Approval can be a starting point for evaluating the benefits of AI application for the health outcomes of patients, which requires higher levels of evidence than what is often needed for regulatory approval [27].”
BackgroundBreast lesion characterization using Quantitative Ultrasound (QUS) and derivative texture methods
Osapoetra, L. O., Sannachi, L., DiCenzo, D., Quiaoit, K., Fatima, K., & Czarnota, G. J. (2020) · Translational Oncology · doi:10.1016/j.tranon.2020.100827 · Cited by 43
“As the number of observations increases, hold-out cross-validation is more relevant for assessing the performance of the diagnostic model to an independent test set that was not used in model development [36].”
Clinical applications of artificial intelligence in liver imaging
Yamada, A., Kamagata, K., Hirata, K., Ito, R., Nakaura, T., Ueda, D., Fujita, S., Fushimi, Y., Fujima, N., Matsui, Y., et al. (2023) · La radiologia medica · doi:10.1007/s11547-023-01638-1 · Cited by 42
The Role of Artificial Intelligence in the Identification and Evaluation of Bone Fractures
Tieu, A., Kroen, E., Kadish, Y., Liu, Z., Patel, N. R., Zhou, A., Yilmaz, A., Lee, S., & Deyer, T. W. (2024) · Bioengineering · doi:10.3390/bioengineering11040338 · Cited by 42
“Lack of standardization across datasets also leads to intrinsic model biases depending on the geographic locations, pathologies, and imaging modalities represented in a given dataset [120,121].”
BackgroundKim, J. H., Han, S. G., Cho, A., Shin, H. J., & Baek, S.-E. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01679-4 · Cited by 38
Park, J. H., Kim, E. Y., Luchini, C., Eccher, A., Tizaoui, K., Shin, J. I., & Lim, B. J. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms23052462 · Cited by 42
“It is also crucial to secure an independent external validation set in model evaluation [66], distinguishing training groups and validation-groups.”
BackgroundAllahabadi, H., Amann, J., Balot, I., Beretta, A., Binkley, C. E., Bozenhard, J., Bruneault, F., Brusseau, J., Candemir, S., Cappellini, L. A., et al. (2022) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2022.3195114 · Cited by 42
“It would need a large dataset with diverse, high-quality images curated from multiple institutions and different geographic areas in order to claim and ensure the generalizability of an AI system intended for clinical usage [19], [20].”
BackgroundAgrawal, A., Khatri, G. D., Khurana, B., Sodickson, A. D., Liang, Y., & Dreizin, D. (2023) · Emergency Radiology · doi:10.1007/s10140-023-02121-0 · Cited by 41
“Several position papers on foundational and clinical-translational research have emphasized the importance of AI/ ML algorithm trustworthiness both in terms of transparency of the output, and transparency in methodology [32–34].”
BackgroundWang, Y., Yan, F., Lu, X., Zheng, G., Zhang, X., Wang, C., Zhou, K., Zhang, Y., Li, H., Zhao, Q., et al. (2019) · EBioMedicine · doi:10.1016/j.ebiom.2019.05.040 · Cited by 41
Park, S. H., Kim, Y.-H., Lee, J. Y., Yoo, S., & Kim, C. J. (2019) · Science Editing · doi:10.6087/kcse.164 · Cited by 30
Self-citationApplication of artificial intelligence in gastrointestinal endoscopy
He, Y. S., Su, J. R., Li, Z., Zuo, X. L., & Li, Y. Q. (2019) · Journal of Digestive Diseases · doi:10.1111/1751-2980.12827 · Cited by 41
Applications of Machine Learning in Bone and Mineral Research
Kong, S. H., & Shin, C. S. (2021) · Endocrinology and Metabolism · doi:10.3803/enm.2021.1111 · Cited by 41
“for Reporting of Diagnostic Accuracy Studies–Artificial Intelligence (STARD-AI) guidelines for diagnostic accuracy [73], and others [74,75].”
MethodChae, K. J., Jin, G. Y., Ko, S. B., Wang, Y., Zhang, H., Choi, E. J., & Choi, H. (2020) · Academic Radiology · doi:10.1016/j.acra.2019.05.018 · Cited by 40
Maleš, I., Boban, Z., Kumrić, M., Vrdoljak, J., Berkovic, K., Pogorelić, Z., & Božić, J. (2024) · Scientific Reports · doi:10.1038/s41598-024-63513-x · Cited by 41
Suresh, N. V., Sridhar, J., Selvakumar, A., & Catherine, S. (2024) · Advances in healthcare information systems and administration book series · doi:10.4018/979-8-3693-7452-8.ch001 · Cited by 40
Butnaru, O., Tatarciuc, M., Luchian, I. A., Tudorici, T. A., Balcoș, C. A. M., Budală, D. G., Sîrghe, A., Virvescu, D. I., & Haba, D. (2025) · Medicina · doi:10.3390/medicina61040572 · Cited by 40
“These findings align with previous research, which has reported AI’s ability to standardize radiographic assessments and improve diagnostic reliability [50,51].”
Shaish, H. A., Ahmed, F. S., Lederer, D. J., D’Souza, B. M., Armenta, P. M., Salvatore, M. M., Saqi, A., Huang, S., Jambawalikar, S., & Mutasa, S. (2020) · Annals of the American Thoracic Society · doi:10.1513/annalsats.202001-068oc · Cited by 40
Osapoetra, L. O., Chan, W., Tran, W. T., Kolios, M. C., & Czarnota, G. J. (2020) · PLoS ONE · doi:10.1371/journal.pone.0244965 · Cited by 40
“This was chosen based on the 10% rule of thumb to prevent overfitting [68].”
MethodKulkarni, V., Gawali, M., & Kharat, A. T. (2021) · JMIR Medical Informatics · doi:10.2196/28776 · Cited by 40
Longuefosse, A., Raoult, J., Benlala, I., de Senneville, B. D., Benkert, T., Macey, J., Bui, S., Berger, P., Ferretti, G., Gaubert, J., et al. (2023) · Radiology · doi:10.1148/radiol.230052 · Cited by 40
Yang, Z. P., Gong, J., Li, J., Sun, H., Pan, Y., & Zhao, L. (2023) · International Journal of Surgery · doi:10.1097/js9.0000000000000441 · Cited by 40
“Although the de fi nition of external validation varies between the RQS and TRIPOD statement, it is still preferable to develop EC radiomic models using independent cohorts from multiple centers in future studies [80,81] .”
Narayana, P. A., Coronado, I., Sujit, S. J., Sun, X., Wolinsky, J. S., & Gabr, R. E. (2019) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2019.10.003 · Cited by 39
“In machine learning, it is common to divide the data into three parts: 1) training, 2) validation, and 3) testing [23].”
BackgroundEmergent situations for smart cities: A survey
Al-Smadi, A. M., Alsmadi, M. K., Baareh, A. K., Almarashdeh, I., Abouelmagd, H., & Shidwan, O. S. (2019) · International Journal of Electrical and Computer Engineering (IJECE) · doi:10.11591/ijece.v9i6.pp4777-4787 · Cited by 39
“Relying on a literature review of studies published between 1990 and 2018, the technological revolution influenced everything [17-40], and the use of Artificial Intelligence (AI) algorithms is expansive, particularly in providing solution to challenging problems including analysis of medical image [41-45], nurse roste…”
BackgroundTıp Fakültesi Öğrencilerinin Tıpta Yapay Zeka ile İlgili Düşünceleri
Öcal, E. E., Atay, E., Önsüz, M. F., Algin, F., Çokyiğit, F. K., Kilinç, S., Köse, Ö. S., & Yiğit, F. N. (2020) · DergiPark (Istanbul University) · Cited by 39
Mitsuyama, Y., Matsumoto, T., Tatekawa, H., Walston, S. L., Kimura, T., Yamamoto, A., Watanabe, T., Miki, Y., & Ueda, D. (2023) · The Lancet Healthy Longevity · doi:10.1016/s2666-7568(23)00133-2 · Cited by 39
Gipson, J., Tang, V. D., Seah, J. C. Y., Kavnoudias, H., Zia, A., Lee, R., Mitra, B., & Clements, W. (2022) · British Journal of Radiology · doi:10.1259/bjr.20210979 · Cited by 38
“25,26 Here we had access to a large cohort of patients with a consistent presentation (blunt trauma) but of varying severity and distribution of injury.”
BackgroundClunie, D., Flanders, A. E., Taylor, A. J., Erickson, B., Bialecki, B., Brundage, D., Gutman, D. A., Prior, F., Seibert, J. A., Perry, J. H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.10473 · Cited by 38
Artificial intelligence for ultrasonography: unique opportunities and challenges
Park, S. H. (2021) · Ultrasonography · doi:10.14366/usg.20078 · Cited by 37
Self-citationJin, K. N., Kim, E. Y., Kim, Y. J., Lee, G. P., Kim, H., Oh, S., Kim, Y., Han, J., & Cho, Y. J. (2022) · European Radiology · doi:10.1007/s00330-021-08397-5 · Cited by 37
“Among 26 finally diagnosed lesions, pneumonia was the most common diagnosis ( n = 696 [12%]; Supplementary Table E2).”
BackgroundMartí‐Bonmatí, L., Koh, D., Riklund, K., Bobowicz, M., Roussakis, Y., Vilanova, J. C., Fütterer, J. J., Rimola, J., Mallol, P., Ribas, G., et al. (2022) · Insights into Imaging · doi:10.1186/s13244-022-01220-9 · Cited by 37
Bai, A., Si, M., Xue, P., Qu, Y., & Jiang, Y. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02397-9 · Cited by 37
Khan, S. D., Hoodbhoy, Z., Raja, M. H. R., Kim, J. Y., Hogg, H. D. J., Manji, A. A. A., Gulamali, F., Hasan, A., Shaikh, A., Tajuddin, S., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000514 · Cited by 36
“[22] Reporting adverse events This refers to the mechanism of reporting and studying any potential adverse patient events or near misses that happen as a result of a selected AI tool use.”
BackgroundArtificial intelligence and machine learning in the fields of education, medical, and smart phones
Mogili, U. R., & Mohamed, A. O. (2023) · AIP conference proceedings · doi:10.1063/5.0175660 · Cited by 37
Cho, S. J., Cho, W., Choi, D., Sim, G., Jeong, S. Y., Baik, S. H., Bae, Y. J., Choi, B. S., Kim, J. H., Yoo, S., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-60781-5 · Cited by 36
Artificial Intelligence in Medicine: Beginner's Guide
Park, S. H. (2018) · Journal of the Korean Society of Radiology · doi:10.3348/jksr.2018.78.5.301 · Cited by 36
“의학적 진단 및 예측을 위한 분류(classification) 방식 인공지능 알고리듬의 정확도를 올바로 검증하 기 위해서는, 과적합(overfitting, 알고리듬이 학습데이터 내에 서는 정확하나 학습에 사용되지 않은 외부 데이터에서는 정확도 가 떨어지는 현상), 스펙트럼 효과(spectrum effect), 유병률 효 과(prevalence effect)가 인공지능 알고리듬의 성능 평가에 어떤 영향을 미치고 어떤 바이어스(bias)를 유발할 수 있는지 잘 이해 할 필요가 있으며 이러한 바이어스(bias)를 피하여 정확한 성능 평가를 하는 것이 매우 중요하다(15).”
Self-citationBig Data and Artificial Intelligence: Opportunities and Threats in Electrophysiology
van de Leur, R. R., Boonstra, M., Bagheri, A., Roudijk, R. W., Sammani, A., Taha, K., Doevendans, P. A., van der Harst, P., van Dam, P. M., Hassink, R. J., et al. (2020) · Arrhythmia & Electrophysiology Review · doi:10.15420/aer.2020.26 · Cited by 36
“Diagnostic or prognostic prediction model research, from simple logistic regression to highly sophisticated DNNs, is characterised 86,87 assessed using the development data set through train-test splitting, cross-validation or bootstrapping.”
MethodDas, A., Connell, M., & Khetarpal, S. (2021) · Clinical Imaging · doi:10.1016/j.clinimag.2021.02.038 · Cited by 36
CroSel: Cross Selection of Confident Pseudo Labels for Partial-Label Learning
Tian, S., Wei, H., Wang, Y., & Feng, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01842 · Cited by 36
Mirfendereski, P., Li, G. Y., Pearson, A. T., & Kerr, A. R. (2025) · Frontiers in Oral Health · doi:10.3389/froh.2025.1569567 · Cited by 35
Kwon, G., Ryu, J., Oh, J., Lim, J., Kang, B.-K., Ahn, C., Bae, J., & Lee, D. K. (2020) · Scientific Reports · doi:10.1038/s41598-020-74653-1 · Cited by 35
“In the internal validation test, we used the AUC, highest accuracy, and highest Youden index to measure performance 32 .”
MethodThe Impact of Artificial Intelligence on Firm Performance
Kim, T., Park, Y., & Kim, W. (2022) · Portland International Conference on Management of Engineering and Technology (PICMET) · doi:10.23919/picmet53225.2022.9882634 · Cited by 35
“The range of tasks that AI covers is continuously expanding their applicational area to medical diagnosis and prediction [12], Energy savings [14], music industry [42] and Household chores [11].”
BackgroundRandomized Clinical Trials of Artificial Intelligence in Medicine: Why, When, and How?
Park, S. H., Choi, J., Fournier, L. S., & Vasey, B. (2022) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0834 · Cited by 35
“This approach is increasingly recognized for clinical AI [3-7] from a scientific perspective and because demonstrating improved health outcomes is desirable in value-based healthcare [8], echoing a similar practice in drug development.”
BackgroundSelf-citationGleeson, F. V., Revel, M., Biederer, J., Larici, A. R., Martini, K., Frauenfelder, T., Screaton, N. J., Prosch, H., Snoeckx, A., Sverzellati, N., et al. (2023) · European Radiology · doi:10.1007/s00330-023-09409-2 · Cited by 35
“Blockchain-based electronic medical records would allow decisions and contributions to hybrid radiology reports to be traced back, enabling the identification of which version of an AI algorithm contributed and when [13].”
BackgroundMethodNew trend in artificial intelligence-based assistive technology for thoracic imaging
Yanagawa, M., Ito, R., Nozaki, T., Fujioka, T., Yamada, A., Fujita, S., Kamagata, K., Fushimi, Y., Tsuboyama, T., Matsui, Y., et al. (2023) · La radiologia medica · doi:10.1007/s11547-023-01691-w · Cited by 36
“However, one of the problems of AI is overfitting [25].”
BackgroundHybrid photonic integrated circuits for neuromorphic computing [Invited]
Xu, R., Taheriniya, S., Ovvyan, A. P., Bankwitz, J. R., McRae, L., Jung, E., Brückerhoff‐Plückelmann, F., Bente, I., Lenzini, F., Bhaskaran, H., et al. (2023) · Optical Materials Express · doi:10.1364/ome.502179 · Cited by 35
Deep Learning for Detection of Pulmonary Metastasis on Chest Radiographs
Hwang, E. J., Lee, J. S., Lee, J. H., Lim, W. H., Kim, J. H., Choi, K. S., Choi, T. W., Kim, T., Goo, J. M., & Park, C. M. (2021) · Radiology · doi:10.1148/radiol.2021210578 · Cited by 34
McSweeney, T. P., Tiulpin, A., Saarakkala, S. S., Niinimäki, J. L., Windsor, R., Jamaludin, A., Kadir, T., Karppinen, J. I., & Määttä, J. H. (2022) · Spine · doi:10.1097/brs.0000000000004572 · Cited by 34
Li, B., Ding, S., Song, G., Li, J., & Zhang, Q. (2019) · Journal of Medical Systems · doi:10.1007/s10916-019-1346-x · Cited by 34
Rudie, J. D., Rauschecker, A. M., Xie, L., Wang, J., Duong, M. T., Botzolakis, E. J., Kovalovich, A. M., Egan, J. M., Cook, T. S., Bryan, R. N., et al. (2020) · Radiology Artificial Intelligence · doi:10.1148/ryai.2020190146 · Cited by 34
“These challenges include a vast spectrum of common and rare pathologic conditions encountered in clinical practice (10), integration of relevant clinical information, highly heterogeneous clinical imaging data, scanner variability, and long processing time for traditional image processing methods.”
BackgroundHaller, S., Van Cauter, S., Federau, C., Hedderich, D. M., & Edjlali, M. (2022) · Neuroradiology · doi:10.1007/s00234-021-02890-w · Cited by 33
“This is commonly referred to as spectrum bias or reference dataset bias, which results from an unrealistic distribution of disease severity in a training dataset [13, 14].”
BackgroundStethoNet: Robust Breast Cancer Mammography Classification Framework
Lamprou, C., Katsikari, K., Rahmani, N. A., Hadjileontiadis, L. J., Seghier, M. L., & Shehhi, A. A. (2024) · IEEE Access · doi:10.1109/access.2024.3473010 · Cited by 33
“From a clinical perspective, where the need for extended validations and high generalization performance is increasingly significant [13], our study offers models that have undergone thorough evaluation using three distinct, publicly available datasets.”
BackgroundMethodInfluentialYi, W., Zhao, J., Tang, W., Yin, H., Yu, L., Wang, Y., & Tian, W. Q. (2023) · European Spine Journal · doi:10.1007/s00586-023-07641-4 · Cited by 33
“As an important step for evaluating a deep learning-based model, external validation is necessary for assessing the model’s robustness and generalization, which could avoid the overestimation of model performance caused by overfitting [14].”
BackgroundAgharia, S., Szatkowski, J., Fraval, A., Stevens, J. M., & Zhou, Y. (2023) · Journal of Orthopaedics · doi:10.1016/j.jor.2023.11.063 · Cited by 33
Statistical considerations for testing an AI algorithm used for prescreening lung CT images
Obuchowski, N. A., & Bullen, J. A. (2019) · Contemporary Clinical Trials Communications · doi:10.1016/j.conctc.2019.100434 · Cited by 33
“Parmar et al. [14], Park and Han [2], and Kim et al. [3] discuss the need for adequate sample sizes for both training and validation, locked independent external testing cohorts collected prospectively from preferably multiple sites to avoid biases, proper blinding of testing cohorts during…”
BackgroundInfluentialShen, H., Jin, Z., Chen, Q., Zhang, L., You, J., Zhang, S., & Zhang, B. (2024) · La radiologia medica · doi:10.1007/s11547-024-01796-w · Cited by 32
“AI models could be vulnerable to spectrum bias and overfitting due to their small sample size as well as lack of external validation [15, 16].”
Choi, J. W., Koo, D. L., Kim, D. H., Nam, H., Lee, J. H., Hong, S., & Kim, B. (2024) · SLEEP · doi:10.1093/sleep/zsae184 · Cited by 32
Deep learning for emergency ascites diagnosis using ultrasonography images
Lin, Z., Li, Z., Cao, P., Lin, Y., Liang, F., He, J., & Huang, L. (2022) · Journal of Applied Clinical Medical Physics · doi:10.1002/acm2.13695 · Cited by 32
Qi, W., Wang, Y., Li, C., He, K., Wang, Y., Huang, S., Li, C., Guo, Q., & Hu, J. (2023) · Journal of Affective Disorders · doi:10.1016/j.jad.2023.04.026 · Cited by 32
Grignaffini, F., Simeoni, P., Alisi, A., & Frezza, F. (2024) · Electronics · doi:10.3390/electronics13163174 · Cited by 32
Yuan, J., Siakallis, L. C., Li, H., Brandner, S., Zhang, J., Li, C., Mancini, L., & Bisdas, S. (2024) · BMC Medical Imaging · doi:10.1186/s12880-024-01274-9 · Cited by 31
“3) [64–66].”
Kim, S., Yoon, H., Lee, M., Kim, M., Han, K., Yoon, J. K., Kim, H. C., Shin, J., & Shin, H. J. (2019) · Scientific Reports · doi:10.1038/s41598-019-55536-6 · Cited by 32
Self-citationLi, Z., Chang, D., Zhang, Z.-X., Luo, F., Liu, Q., Zhang, J., Yang, G., & Wu, W. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3420411 · Cited by 31
“4) Clinical Structure Reconstruction: To further validate the effectiveness and accuracy of our proposed DCDS method, we conducted a series of clinical structure reconstruction experiments [53], [54] [55].”
MethodA Justifiable Investment in AI for Healthcare: Aligning Ambition with Reality
Karpathakis, K., Morley, J., & Floridi, L. (2024) · Minds and Machines · doi:10.1007/s11023-024-09692-y · Cited by 31
Lewicki, K., Lee, M. S. A., Cobbe, J., & Singh, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581463 · Cited by 30
“The challenges of representativeness are likely only be exacerbated in more complex domains, such as healthcare diagnostics, where there may be a virtually unlimited number of individual, group, and social factors that can be relevant, from lifestyle and eating habits, to weather and chemical exposure, and particulari…”
BackgroundCho, J., Kim, Y. J., Sunwoo, L., Lee, G. P., Nguyen, T. Q., Cho, S. J., Baik, S. H., Bae, Y. J., Choi, B. S., Jung, C., et al. (2021) · Frontiers in Oncology · doi:10.3389/fonc.2021.739639 · Cited by 30
“This retrospective study adhered to the relevant reporting guidelines (28 – 30).”
Method
Metadata
- Authors
- Park, S. H., & Han, K.
- Affiliations
- Yonsei University (KR)
- Venue
- Radiology
- Published
- 2018-03
- Volume
- 286
- Issue
- 3
- Pages
- 800-809
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2783687327
- Cited by
- 894
- References
- 49
- Citations by year
- 2026: 123 · 2025: 98 · 2024: 96 · 2023: 135 · 2022: 111 · 2021: 121 · 2020: 103 · 2019: 80 · 2018: 25
- FWCI
- 1.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Ministry of Trade, Industry and Energy
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- artificial intelligence in healthcare, deep neural network, medical image diagnosis, prediction model, discrimination and calibration, overfitting, spectrum bias, external validation, disease prevalence, outcomes research, high-dimensional models
- MeSH terms
- Artificial Intelligence, Data Interpretation, Statistical, Diagnostic Imaging, Humans, Image Interpretation, Computer-Assisted, Terminology as Topic, Predictive Value of Tests, Technology Assessment, Biomedical, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.