Methodologic Guide for Evaluating Clinical Performance and Effect of Artificial Intelligence Technology for Medical Diagnosis and Prediction
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
The use of artificial intelligence in medicine is currently an issue of great interest, especially with regard to the diagnostic or predictive analysis of medical images. Adoption of an artificial intelligence tool in clinical practice requires careful confirmation of its clinical utility. Herein, the authors explain key methodology points involved in a clinical evaluation of artificial intelligence technology for use in medicine, especially high-dimensional or overparameterized diagnostic or predictive models in which artificial deep neural networks are used, mainly from the standpoints of clinical epidemiology and biostatistics. First, statistical methods for assessing the discrimination and calibration performances of a diagnostic or predictive model are summarized. Next, the effects of disease manifestation spectrum and disease prevalence on the performance results are explained, followed by a discussion of the difference between evaluating the performance with use of internal and external datasets, the importance of using an adequate external dataset obtained from a well-defined clinical cohort to avoid overestimating the clinical performance as a result of overfitting in high-dimensional or overparameterized classification model and spectrum bias, and the essentials for achieving a more robust clinical evaluation. Finally, the authors review the role of clinical trials and observational outcome studies for ultimate clinical verification of diagnostic or predictive artificial intelligence tools through patient outcomes, beyond performance metrics, and how to design such studies. © RSNA, 2018
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This common scenario means that estimations of AI performance based on internal test data may overestimate its real-world performance on external data [73–75].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [74]
In the citing paper
“Similar to internal validation of the AIPM, external AIPM model validation can be based on discrimination (area under the receiver operating characteristic curve, sensitivity, specificity, positive and negative predictive values), calibration (calibration plot)5,6,13,17,49,56,78, and Decision Curve Analysis5,6,17,49,68,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“To ensure the generalizability of the AIPM to the intended healthcare setting, developers are advised to extensively validate the model for representative data from that setting6–8,10,13–15,24,27,56,64,66,68,72,77,79,87,88 (see phase 1, Representativeness).”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“The literature recommends to assess generalizability on external data from a different time period, place, or healthcare setting3,7,8,10,16,17,27,56,68,72,79.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“Although prospective validation is rare, it is preferred by the literature5,14,56, as it provides a better idea of the AIPM’s true applicability to medical practice and allows the healthcare professionals to identify and review errors in real time19,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
(2016) · Cited by 11,679
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293
Lakhani, P., & Sundaram, B. (2017) · Radiology · doi:10.1148/radiol.2017162326 · Cited by 1,775
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Radiology · doi:10.1148/radiol.2015151516 · Cited by 1,337
Deep Learning in Medical Imaging: General Overview
Lee, J.-G., Jun, S., Cho, Y.-W., Lee, H., Kim, G. B., Seo, J. B., & Kim, N. (2017) · Korean Journal of Radiology · doi:10.3348/kjr.2017.18.4.570 · Cited by 1,276
Research works printed but not linked: 3
- Steyerberg EW, 2010, Clinical prediction models: a practical approach to development, validation, and updating, 83
- Steyerberg EW, 2010, Clinical prediction models: a practical approach to development, validation, and updating, 299
- Newman TB, 2013, Designing clinical research, 117
Cited by
Convolutional neural networks: an overview and application in radiology
Yamashita, R., Nishio, M., Do, R. K. G., & Togashi, K. (2018) · Insights into Imaging · doi:10.1007/s13244-018-0639-9 · Cited by 4,682
“It is worthy of mention that the term Bvalidation^ is used differently in the medical field and the machine learning field [26].”
BackgroundChecklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443
Preparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020
“This is especially true in the clinical setting and is well outlined by Park and Han (22).”
BackgroundArtificial intelligence for mental health and mental illnesses: an overview
Graham, S., Depp, C., Lee, E. E., Nebeker, C., Tu, X., Kim, H.-C., & Jeste, D. V. (2019) · Current Psychiatry Reports · doi:10.1007/s11920-019-1094-0 · Cited by 970
Lekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957
“Initial efforts have focused on providing recommendations for the reporting of AI studies for different medical domains or clinical tasks (eg, TRIPOD+AI, 11 CLAIM, 12 CONSORT-AI, 13 DECIDE-AI, 14 PROBAST-AI, 15 CLEAR 16 ).”
Predicting cancer outcomes with radiomics and artificial intelligence in radiology
Bera, K., Braman, N., Gupta, A., Velcheti, V., & Madabhushi, A. (2022) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-021-00560-7 · Cited by 819
“Data from retrospectively acquired data sets are often most convenient to aggregate, but raise challenges related to data purity for both model training and validation because predefined inclusion and exclusion criteria might result in unconscious biases in AI algorithms 144 .”
BackgroundMachine learning for medical imaging: methodological failures and recommendations for the future
Varoquaux, G., & Cheplygina, V. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00592-y · Cited by 656
Fairness of artificial intelligence in healthcare: review and recommendations
Ueda, D., Kakinuma, T., Fujita, S., Kamagata, K., Fushimi, Y., Ito, R., Matsui, Y., Nozaki, T., Nakaura, T., Fujima, N., et al. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01474-3 · Cited by 607
Heller, N., Isensee, F., Maier-Hein, K. H., Hou, X., Xie, C., Li, F., Nan, Y., Mu, G., Lin, Z., Han, M., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101821 · Cited by 579
Convolutional Neural Networks for Radiologic Images: A Radiologist’s Guide
Soffer, S., Ben-Cohen, A., Shimon, O., Amitai, M. M., Greenspan, H., & Klang, E. (2019) · Radiology · doi:10.1148/radiol.2018180547 · Cited by 577
Deep learning for tomographic image reconstruction
Wang, G., Ye, J. C., & De Man, B. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00273-z · Cited by 531
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499
“…of the AIPM, external AIPM model validation can be based on discrimination (area under the receiver operating characteristic curve, sensitivity, speci fi city, positive and negative predictive values), calibration (calibration plot) 5,6,13,17,49,56,78 , and Decision Curve Analysis 5,6,17,49,68,72 .”
BackgroundMethodInfluentialKim, D. W., Jang, H. Y., Kim, K. W., Shin, Y., & Park, S. H. (2019) · Korean Journal of Radiology · doi:10.3348/kjr.2019.0025 · Cited by 459
“Therefore, individuals developing such AI algorithms might rely on whatever data are available (methodologically referred to as convenience case-control data), although these may be prone to selection biases and artificial disease prevalence and likely not represent real-world clinical settings well (12, 19, 20).”
BackgroundMethodSelf-citationLee, E. E., Torous, J., De Choudhury, M., Depp, C. A., Graham, S. A., Kim, H.-C., Paulus, M. P., Krystal, J. H., & Jeste, D. V. (2021) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2021.02.001 · Cited by 429
“Temporal validation (newly recruited participants) and geographical validation (data from a different site) are more rigorous strategies to ensure clinical utility (see (87,88)).”
BackgroundExternal Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review
Yu, A. C., Mohajer, B., & Eng, J. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.210064 · Cited by 408
Aeffner, F., Zarella, M. D., Buchbinder, N., Bui, M. M., Goodman, M. R., Hartman, D. J., Lujan, G. M., Molani, M. A., Parwani, A. V., Lillard, K., et al. (2019) · Journal of Pathology Informatics · doi:10.4103/jpi.jpi_82_18 · Cited by 403
Park, J. E., Park, S. Y., Kim, H. J., & Kim, H. S. (2019) · Korean Journal of Radiology · doi:10.3348/kjr.2018.0070 · Cited by 388
“Differing prevalence does not affect the diagnostic performance such as area under the receiver operating characteristics curve, sensitivity, and specificity, but it does affect positive and negative predictive values (76, 77).”
BackgroundSelf-citationHung, K., Montalvao, C., Tanaka, R., Kawai, T., & Bornstein, M. M. (2020) · Dentomaxillofacial Radiology · doi:10.1259/dmfr.20190107 · Cited by 387
“82 86 The diagnostic performance of the models in these pre-clinical studies exhibited satisfactory results, but these results might be overly optimistic due to the training-testing images presenting obvious lesions from extracted teeth and lacking the image of other oral tissues.”
Breast Cancer Prediction: A Comparative Study Using Machine Learning Techniques
Islam, M. M., Haque, M. R., Iqbal, H., Hasan, M. M., Hasan, M., & Kabir, M. N. (2020) · SN Computer Science · doi:10.1007/s42979-020-00305-w · Cited by 361
“A significant issue in the field of bioinformatics or medical science [1] is the accurate diagnosis of certain important information.”
BackgroundBluemke, D. A., Moy, L., Bredella, M. A., Ertl-Wagner, B. B., Fowler, K. J., Goh, V. J., Halpern, E. F., Hess, C. P., Schiebler, M. L., & Weiss, C. R. (2020) · Radiology · doi:10.1148/radiol.2019192515 · Cited by 343
Freeman, K., Geppert, J., Stinton, C., Todkill, D., Johnson, S., Clarke, A., & Taylor-Phillips, S. (2021) · BMJ · doi:10.1136/bmj.n1872 · Cited by 338
“42 For screening, however, temporal validation could introduce bias because, for instance, the same women might attend repeat screens, and be screened by the same personnel using the same machines.”
BackgroundMethodInfluentialBalki, I., Amirabadi, A., Levman, J., Martel, A. L., Emersic, Z., Meden, B., Garcia-Pedrero, A., Ramirez, S. C., Kong, D., Moody, A. R., et al. (2019) · Canadian Association of Radiologists Journal · doi:10.1016/j.carj.2019.06.002 · Cited by 299
“Position statements by radiologic associations have highlighted that adequate sample size considerations are essential to ensure robust, unbiased clinical model development [11,14].”
BackgroundPresenting machine learning model information to clinical end users with model facts labels
Sendak, M. P., Gao, M., Brajer, N., & Balu, S. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0253-3 · Cited by 279
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
“In [129, 130, 131, 132, 133, 134], the authors aim to cover many facets of XAI technologies ranging from problem definitions, goals, AI/ML model explanations to evaluation measures, while in [135] the authors emphasize the research trends and challenges in Human-Computer-Interaction (HCI) applications.”
BackgroundMajkowska, A., Mittal, S., Steiner, D. F., Reicher, J. J., McKinney, S. M., Duggan, G. E., Eswaran, K., Cameron Chen, P.-H., Liu, Y., Kalidindi, S. R., et al. (2020) · Radiology · doi:10.1148/radiol.2019191293 · Cited by 277
“However, because such data do not necessarily reflect real-world prevalence or diversity (33,34), such enrichment can also prevent meaningful clinical interpretation of diagnostic performance.”
BackgroundSchachner, T., Keller, R., & v Wangenheim, F. (2020) · Journal of Medical Internet Research · doi:10.2196/20701 · Cited by 267
“In addition to the above-mentioned standardized guidelines for research in health care, future research should make use of already existing guidelines for reporting the technical part of AI-based conversational agents used in health care and medicine [72,73].”
BackgroundApplication of artificial intelligence in gastroenterology
Yang, Y. J., & Bang, C. S. (2019) · World Journal of Gastroenterology · doi:10.3748/wjg.v25.i14.1666 · Cited by 263
Qian, X., Pei, J., Zheng, H., Xie, X., Yan, L., Zhang, H., Han, C., Gao, X., Zhang, H., Zheng, W., et al. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-021-00711-2 · Cited by 253
“To strengthen the generalizability of the AI system, it must not only be used to evaluate retrospective data, but also it should be assessed in a prospective study setting 36,37 .”
Sollini, M., Antunovic, L., Chiti, A., & Kirienko, M. (2019) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-019-04372-x · Cited by 250
Sim, Y., Chung, M. J., Kotter, E., Yune, S., Kim, M., Do, S., Han, K., Kim, H., Yang, S., Lee, D.-J., et al. (2020) · Radiology · doi:10.1148/radiol.2019182465 · Cited by 248
Self-citationvan Velzen, S. G. M., Lessmann, N., Velthuis, B. K., Bank, I. E. M., van den Bongard, D. H. J. G., Leiner, T., de Jong, P. A., Veldhuis, W. B., Correa, A., Terry, J. G., et al. (2020) · Radiology · doi:10.1148/radiol.2020191621 · Cited by 244
“Despite many efforts supporting the development of robust deep learning (DL) software (16,17), an established limitation of DL algorithms is that performance can rapidly degrade and fail with what appears to humans as only slight variations in the input data (18).”
BackgroundEthical considerations in artificial intelligence
Safdar, N. M., Banja, J. D., & Meltzer, C. C. (2020) · European Journal of Radiology · doi:10.1016/j.ejrad.2019.108768 · Cited by 243
Deep Learning for Diagnosis of Chronic Myocardial Infarction on Nonenhanced Cardiac Cine MRI
Zhang, N., Yang, G., Gao, Z., Xu, C., Zhang, Y., Shi, R., Keegan, J., Xu, L., Zhang, H., Fan, Z., et al. (2019) · Radiology · doi:10.1148/radiol.2019182304 · Cited by 231
Artificial Intelligence in Low- and Middle-Income Countries: Innovating Global Health Radiology
Mollura, D. J., Culp, M. P., Pollack, E., Battino, G., Scheel, J. R., Mango, V. L., Elahi, A., Schweitzer, A., & Dako, F. (2020) · Radiology · doi:10.1148/radiol.2020201434 · Cited by 228
“Clinical education for radiologists and technologists in LMICs can reinforce the skills necessary for critically appraising AI outputs and investigating AI’s clinical utility, performance, and result justifications (8).”
BackgroundEthics of Artificial Intelligence in Medicine and Ophthalmology
Abdullah, Y. I., Schuman, J. S., Shabsigh, R., Caplan, A., & Al-Aswad, L. A. (2021) · Asia-Pacific Journal of Ophthalmology · doi:10.1097/apo.0000000000000397 · Cited by 151
“Here are some approaches to the issues of bias: • Robust data sources fed into algorithms; • Sometimes, accepting real data with their implicit bias; 39,44 • Thoroughly examining training data for bias; 27”
BackgroundPeng, J., Kang, S., Ning, Z., Deng, H., Shen, J., Xu, Y., Zhang, J., Zhao, W., Li, X., Gong, W., et al. (2020) · European Radiology · doi:10.1007/s00330-019-06318-1 · Cited by 218
Artificial intelligence revolutionizing drug development: Exploring opportunities and challenges
Tiwari, P. C., Pal, R., Chaudhary, M. J., & Nath, R. (2023) · Drug Development Research · doi:10.1002/ddr.22115 · Cited by 219
“AI can further be utilized to design more effective clinical trials, resulting in a substantial reduction of time and costs associated with bringing a new drug to market (Fountzilas et al., 2022; Park & Han, 2018).”
BackgroundGraham, S. A., Lee, E. E., Jeste, D. V., Van Patten, R., Twamley, E. W., Nebeker, C., Yamada, Y., Kim, H.-C., & Depp, C. A. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2019.112732 · Cited by 211
Assessment of Convolutional Neural Networks for Automated Classification of Chest Radiographs
Dunnmon, J. A., Yi, D., Langlotz, C. P., Ré, C., Rubin, D. L., & Lungren, M. P. (2019) · Radiology · doi:10.1148/radiol.2018181422 · Cited by 206
Current applications and development of artificial intelligence for digital dental radiography
Putra, R. H., Doi, C., Yoda, N., Astuti, E. R., & Sasaki, K. (2022) · Dentomaxillofacial Radiology · doi:10.1259/dmfr.20210197 · Cited by 203
The current state of artificial intelligence in ophthalmology
Kapoor, R., Walters, S. P., & Al-Aswad, L. A. (2019) · Survey of Ophthalmology · doi:10.1016/j.survophthal.2018.09.002 · Cited by 198
What should medical students know about artificial intelligence in medicine?
Park, S. H., Do, K.-H., Kim, S., Park, J. H., & Lim, Y.-S. (2019) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2019.16.18 · Cited by 196
“For proper validation of the clinical accuracy of AI algorithms, the test data should have the following features [11,16,17]: (1) data collected avoiding biases from specific indications (i.”
BackgroundMethodSelf-citationQuantitative Phase Imaging and Artificial Intelligence: A Review
Jo, Y., Cho, H., Lee, S. Y., Choi, G., Kim, G., Min, H.-S., & Park, Y. (2019) · IEEE Journal of Selected Topics in Quantum Electronics · doi:10.1109/jstqe.2018.2859234 · Cited by 192
Radiomics and Its Feature Selection: A Review
Zhang, W., Guo, Y. T., & Jin, Q. (2023) · Symmetry · doi:10.3390/sym15101834 · Cited by 192
How to develop a meaningful radiomic signature for clinical use in oncologic patients
Papanikolaou, N., Matos, C., & Koh, D. M. (2020) · Cancer Imaging · doi:10.1186/s40644-020-00311-4 · Cited by 186
Hagiwara, A., Fujita, S., Ohno, Y., & Aoki, S. (2020) · Investigative Radiology · doi:10.1097/rli.0000000000000666 · Cited by 183
Artificial Intelligence for Medical Image Analysis: A Guide for Authors and Reviewers
England, J. R., & Cheng, P. M. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.18.20490 · Cited by 180
Sajjadian, M., Lam, R. W. M., Milev, R. V., Rotzinger, S., Frey, B. N., Soares, C. N., Parikh, S. V., Foster, J. A., Turecki, G., Müller, D. J., et al. (2021) · Psychological Medicine · doi:10.1017/s0033291721003871 · Cited by 180
“In addition, the essential role of using an external validation dataset to prevent overfitting in high-dimensional classification algorithms should be taken into consideration (Park & Han, 2018).”
BackgroundMartín Noguerol, T., Paulano-Godino, F., Martín-Valdivia, M. T., Menias, C. O., & Luna, A. (2019) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2019.05.047 · Cited by 175
Dzobo, K., Adotey, S. K., Thomford, N. E., & Dzobo, W. (2019) · OMICS A Journal of Integrative Biology · doi:10.1089/omi.2019.0038 · Cited by 174
“AI technology’s ability to analyze and offer a diagnosis based on image analysis is already being utilized in clinics across the globe (Dande and Samant, 2018; Das et al., 2018; Park and Han, 2018).”
Park, S. Y., Park, J. E., Kim, H., & Park, S. H. (2021) · Korean Journal of Radiology · doi:10.3348/kjr.2021.0223 · Cited by 176
“org the conventional ROC analysis) is sometimes referred to as the ‘C index’ in the literature [2,21,26].”
BackgroundMethodInfluentialSelf-citationConvolutional Neural Network in Medical Image Analysis: A Review
Kshatri, S. S., & Singh, D. (2023) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-023-09898-w · Cited by 172
“Where appropriate, we have defined the term validation [120].”
BackgroundReimagining Healthcare: Unleashing the Power of Artificial Intelligence in Medicine
Iqbal, J., Cortés Jaimes, D. C., Makineni, P., Subramani, S., Hemaida, S., Thugu, T. R., Butt, A. N., Sikto, J. T., Kaur, P., Lak, M. A., et al. (2023) · Cureus · doi:10.7759/cureus.44658 · Cited by 94
Implementation of Clinical Artificial Intelligence in Radiology: Who Decides and How?
Daye, D., Wiggins, W. F., Lungren, M. P., Alkasab, T., Kottler, N., Allen, B., Roth, C. J., Bizzo, B. C., Durniak, K., Brink, J. A., et al. (2022) · Radiology · doi:10.1148/radiol.212151 · Cited by 138
Deep Learning: An Update for Radiologists
Cheng, P. M., Montagnon, E., Yamashita, R., Pan, I., Cadrin-Chênevert, A., Perdigón Romero, F., Chartrand, G., Kadoury, S., & Tang, A. (2021) · RadioGraphics · doi:10.1148/rg.2021200210 · Cited by 166
“Such an assessment may include temporal validation with newly recruited patients, or geographic validation with data from a different site (67).”
Deep Learning for Chest Radiograph Diagnosis in the Emergency Department
Hwang, E. J., Nam, J. G., Lim, W. H., Park, S. J., Jeong, Y. S., Kang, J. H., Hong, E. K., Kim, T. M., Goo, J. M., Park, S., et al. (2019) · Radiology · doi:10.1148/radiol.2019191225 · Cited by 164
Self-citationArtificial Intelligence and Health Technology Assessment: Anticipating a New Level of Complexity
Alami, H., Lehoux, P., Auclair, Y., de Guise, M., Gagnon, M.-P., Shaw, J., Roy, D., Fleet, R., Ag Ahmed, M. A., & Fortin, J.-P. (2020) · Journal of Medical Internet Research · doi:10.2196/17707 · Cited by 164
“At present, it is difficult to distinguish the effect of an AI-based decision from the overall preventive and/or therapeutic strategy of patient care [49,50].”
BackgroundEche, T., Schwartz, L. H., Mokrane, F.-Z., & Dercle, L. (2021) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2021210097 · Cited by 163
Methods for clinical evaluation of artificial intelligence algorithms for medical diagnosis
Park, S. H., Han, K., Jang, H. Y., Park, J. E., Lee, J.-G., Kim, D. W., & Choi, J. (2023) · Radiology · doi:10.1148/radiol.220182 · Cited by 163
Self-citationA Clinician's Guide to Artificial Intelligence: How to Critically Appraise Machine Learning Studies
Faes, L., Liu, X., Wagner, S. K., Fu, D. J., Balaskas, K., Sim, D. A., Bachmann, L. M., Keane, P. A., & Denniston, A. K. (2020) · Translational Vision Science & Technology · doi:10.1167/tvst.9.2.7 · Cited by 162
Pacilè, S., Lopez, J., Chone, P., Bertinotti, T., Grouin, J. M., & Fillard, P. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020190208 · Cited by 161
“This type of validation is often referred to as external geographic validation and shall be preferred to other types of validation when evaluating generalizability of AI models (28).”
MethodDecentralized federated learning through proxy model sharing
Kalra, S., Wen, J., Cresswell, J. C., Volkovs, M., & Tizhoosh, H. R. (2023) · Nature Communications · doi:10.1038/s41467-023-38569-4 · Cited by 160
Deep Learning Enables Automatic Classification of Emphysema Pattern at CT
Humphries, S. M., Notary, A. M., Centeno, J. P., Strand, M. J., Crapo, J. D., Silverman, E. K., Lynch, D. A., & For the Genetic Epidemiology of COPD (COPDGene) Investigators (2020) · Radiology · doi:10.1148/radiol.2019191022 · Cited by 160
“Chronic bronchitis 77 (10) 380 (14) 331 (21) 295 (24) 199 (26) 43 (26) ,.”
BackgroundInfluentialRadiomics and Deep Learning: Hepatic Applications
Park, H. J., Park, B., & Lee, S. S. (2020) · Korean Journal of Radiology · doi:10.3348/kjr.2019.0752 · Cited by 159
“…acid-enhanced arterial phase MR images using a CNN algorithm. kjronline.org and split-sample validation (i.e., splitting the entire dataset randomly into training and the validation data) may not sufficiently guarantee the generalizability of radiomics models or deep learning algorithms (70, 71).”
MethodEvaluation of machine learning algorithms for health and wellness applications: A tutorial
Tohka, J., & van Gils, M. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104324 · Cited by 131
Multitask Deep Learning for Segmentation and Classification of Primary Bone Tumors on Radiographs
von Schacky, C. E., Wilhelm, N., Schäfer, V. S., Leonhardt, Y., Gassert, F. G., Foreman, S. C., Gassert, F. T., Jung, M., Jungmann, P. M., Russe, M. F., et al. (2021) · Radiology · doi:10.1148/radiol.2021204531 · Cited by 160
A two-stage deep learning architecture for radiographic staging of periodontal bone loss
Jiang, L., Chen, D., Cao, Z., Wu, F., Zhu, H., & Zhu, F. (2022) · BMC Oral Health · doi:10.1186/s12903-022-02119-z · Cited by 159
“Furthermore, the model was not externally verified, which may lead to overfitting to the training data set, potentially resulting in an overestimation of the model’s performance [27, 28].”
Park, S. H., Choi, J., & Byeon, J. (2021) · Korean Journal of Radiology · doi:10.3348/kjr.2021.0048 · Cited by 152
“일반화에 대한 취약성이 갖는 의미 먼저, 인공지능 알고리즘의 성능 검증에 있어 이 논문의 뒤에서 설명하는 외부데이터를 이용한 외부검증이 매우 중 요하다[6,10,11,21-29].”
InfluentialSelf-citationStructural and functional radiomics for lung cancer
Wu, G., Jochems, A., Refaee, T., Ibrahim, A., Yan, C., Sanduleanu, S., Woodruff, H. C., & Lambin, P. (2021) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-021-05242-1 · Cited by 151
Deep learning-based approach to diagnose lung cancer using CT-scan images
Shatnawi, M. Q., Abuein, Q., & Al-Quraan, R. (2024) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2024.100188 · Cited by 151
Quality assurance of computer-aided detection and diagnosis in colonoscopy
Vinsard, D. G., Mori, Y., Misawa, M., Kudo, S.-E., Rastogi, A., Bagci, U., Rex, D. K., & Wallace, M. B. (2019) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2019.03.019 · Cited by 137
rs-fMRI and machine learning for ASD diagnosis: a systematic review and meta-analysis
Santana, C. P., Carvalho, E., Rodrigues, I. D., Bastos, G. S., de Souza, A. D., & de Brito, L. L. (2022) · Scientific Reports · doi:10.1038/s41598-022-09821-6 · Cited by 149
“…the studies with low RoB between the ones that performed a temporal (using data from newly recruited subjects) or geographic (using data collected by independent investigators at a different site) validation (5 articles and 35 samples) and the ones that performed a split-sample validation 127 .”
MethodArtificial intelligence and colonoscopy: Current status and future perspectives
Kudo, S., Mori, Y., Misawa, M., Takeda, K., Kudo, T., Itoh, H., Oda, M., & Mori, K. (2019) · Digestive Endoscopy · doi:10.1111/den.13340 · Cited by 147
Lee, J. H., Joo, I., Kang, T. W., Paik, Y. H., Sinn, D. H., Ha, S. Y., Kim, K., Choi, C., Lee, G., Yi, J., et al. (2020) · European Radiology · doi:10.1007/s00330-019-06407-1 · Cited by 144
“Fifth, to ensure that the over-fitting issue common in deep learning model development did not occur [39], we tested an alternative clinical set (the external test set) with a different distribution of liver fibrosis.”
BackgroundTsopra, R., Fernandez, X., Luchinat, C., Alberghina, L., Lehrach, H., Vanoni, M., Dreher, F., Sezerman, O. U., Cuggia, M., de Tayrac, M., et al. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01634-3 · Cited by 116
Xie, Y., Chen, M., Kao, D., Gao, G., & Chen, X. '. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376807 · Cited by 141
van de Sande, D., Van Genderen, M. E., Smit, J. M., Huiskens, J., Visser, J. J., Veen, R. E. R., van Unen, E., BA, O. H., Gommers, D., & Bommel, J. V. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100495 · Cited by 135
Artificial intelligence for biomarker discovery in Alzheimer's disease and dementia
Winchester, L. M., Harshfield, E. L., Shi, L., Badhwar, A., Khleifat, A. A., Clarke, N., Dehsarvi, A., Lengyel, I., Lourida, I., Madan, C. R., et al. (2023) · Alzheimer's & Dementia · doi:10.1002/alz.13390 · Cited by 130
“104,105 Thisreliesonanumberoffactors, including health service infrastructure and resources, as well as trust from stakeholders in AI.”
BackgroundBrain metastasis detection using machine learning: a systematic review and meta-analysis
Cho, S. J., Sunwoo, L., Baik, S. H., Bae, Y. J., Choi, B. S., & Kim, J. H. (2021) · Neuro-Oncology · doi:10.1093/neuonc/noaa232 · Cited by 115
Training Strategies for Radiology Deep Learning Models in Data-limited Scenarios
Candemir, S., Nguyen, X. V., Folio, L. R., & Prevedello, L. M. (2021) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2021210014 · Cited by 102
Fritz, B., Yi, P. H., Kijowski, R., & Fritz, J. M. (2022) · Investigative Radiology · doi:10.1097/rli.0000000000000907 · Cited by 127
Twilt, J. J., van Leeuwen, K. G., Huisman, H. J., Fütterer, J. J., & de Rooij, M. (2021) · Diagnostics · doi:10.3390/diagnostics11060959 · Cited by 97
“67, indicating the importance of external validation for robust performance assessment [33,49].”
BackgroundDeep Learning in Diagnosis of Maxillary Sinusitis Using Conventional Radiography
Kim, Y., Lee, K. J., Sunwoo, L., Choi, D., Nam, C.-M., Cho, J., Kim, J., Bae, Y. J., Yoo, R.-E., Choi, B. S., et al. (2019) · Investigative Radiology · doi:10.1097/rli.0000000000000503 · Cited by 109
Jarrahi, M. H. (2019) · Business Information Review · doi:10.1177/0266382119883999 · Cited by 83
“A fertile ground for this is medical diagnosis, in which AI will be able to support physicians (e.g. radiologists) and augment their diagnostic skills (Park and Han, 2018).”
BackgroundChoi, J. W., Cho, Y. J., Lee, S., Lee, J. S., Lee, S., Choi, Y. H., Cheon, J., & Ha, J. Y. (2019) · Investigative Radiology · doi:10.1097/rli.0000000000000615 · Cited by 119
“An external test, more commonly referred to as external validation in other medical fields,(18) is an important means of preventing the overly optimistic diagnostic performance of a predictive model that results from overfitting.(17) To the best of our knowledge, our study is the first in pediatricmusculoskeletal radi…”
BackgroundMethodInfluentialArtificial intelligence in radiology
Filipović-Grčić, L., & Đerke, F. (2019) · Rad Hrvatske akademije znanosti i umjetnosti Medicinske znanosti · doi:10.21857/y26kec3o79 · Cited by 118
Voter, A. F., Larson, M. E., Garrett, J. W., & Yu, J.-P. J. (2021) · American Journal of Neuroradiology · doi:10.3174/ajnr.a7179 · Cited by 100
“AI algorithms are known to have numerous limitations, including the need for large, diverse, and unbiased datasets, 14 which can be difficult to acquire or curate 15 and operate in a manner that precludes direct interrogation of the decision process itself.”
BackgroundMitigating Bias in Radiology Machine Learning: 3. Performance Metrics
Faghani, S., Khosravi, B., Zhang, K., Moassefi, M., Jagtap, J. M., Nugen, F., Vahdati, S., Kuanar, S. P., Rassoulinejad-Mousavi, S. M., Singh, Y., et al. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220061 · Cited by 94
“Although these techniques can help determine model fitness, internal testing may not detect biases present in the original data (19,20).”
BackgroundYusuf, M., Atal, I., Li, J., Smith, P., Ravaud, P., Fergie, M., Callaghan, M., & Selfe, J. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-034568 · Cited by 102
A case-based ensemble learning system for explainable breast cancer recurrence prediction
Gu, D., Su, K., & Zhao, H. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101858 · Cited by 113
A scalable physician-level deep learning algorithm detects universal trauma on pelvic radiographs
Cheng, C., Wang, Y., Chen, H., Hsiao, P.-M., Yeh, C., Hsieh, C., Miao, S., Xiao, J. Z., Liao, C., & Lü, L. (2021) · Nature Communications · doi:10.1038/s41467-021-21311-3 · Cited by 113
Behzadi-khormouji, H., Rostami, H., Salehi, S., Derakhshande-Rishehri, T., Masoumi, M., Salemi, S., Keshavarz, A., Gholamrezanezhad, A., Assadi, M., & Batouli, A. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.105162 · Cited by 112
Artificial Intelligence in Breast Imaging: Potentials and Limitations
Mendelson, E. B. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.18.20532 · Cited by 106
Artificial Intelligence in Imaging: The Radiologist’s Role
Rubin, D. L. (2019) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2019.05.036 · Cited by 110
Hickman, S. E., Baxter, G. C., & Gilbert, F. J. (2021) · British Journal of Cancer · doi:10.1038/s41416-021-01333-w · Cited by 110
Dupuis, M., Delbos, L., Veil, R., & Adamsbaum, C. (2022) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2021.10.007 · Cited by 94
A Review on the Current Applications of Artificial Intelligence in the Operating Room
Birkhoff, D. C., van Dalen, A. S. H. M., & Schijven, M. P. (2021) · Surgical Innovation · doi:10.1177/1553350621996961 · Cited by 90
“Park et al. 53 acknowledged the problem of irregular research designs in medical AI studies.”
Early predictive model for breast cancer classification using blended ensemble learning
Mahesh, T. R., Kumar, V. V., Vivek, V., Raghunath, K. M. K., & Sindhu Madhuri, G. (2022) · International Journal of Systems Assurance Engineering and Management · doi:10.1007/s13198-022-01696-0 · Cited by 107
“The proper diagnosis of certain crucial information is a remarkable issue in the area of bioinformatics or medical research (Park and Han 2018).”
Reismann, J., Romualdi, A., Kiss, N., Minderjahn, M. I., Kallarackal, J., Schad, M., & Reismann, M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0222030 · Cited by 105
Zhang, J., Yao, K., Liu, P., Liu, Z., Han, T., Zhao, Z., Cao, Y., Zhang, J., Zhang, J., Tian, J., et al. (2020) · EBioMedicine · doi:10.1016/j.ebiom.2020.102933 · Cited by 105
Metadata
- Authors
- Park, S. H., & Han, K.
- Affiliations
- Yonsei University (KR)
- Venue
- Radiology
- Published
- 2018-03
- Volume
- 286
- Issue
- 3
- Pages
- 800-809
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2783687327
- Cited by
- 894
- References
- 49
- Citations by year
- 2026: 123 · 2025: 98 · 2024: 96 · 2023: 135 · 2022: 111 · 2021: 121 · 2020: 103 · 2019: 80 · 2018: 25
- FWCI
- 1.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Ministry of Trade, Industry and Energy
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- artificial intelligence in healthcare, deep neural network, medical image diagnosis, prediction model, discrimination and calibration, overfitting, spectrum bias, external validation, disease prevalence, outcomes research, high-dimensional models
- MeSH terms
- Artificial Intelligence, Data Interpretation, Statistical, Diagnostic Imaging, Humans, Image Interpretation, Computer-Assisted, Terminology as Topic, Predictive Value of Tests, Technology Assessment, Biomedical, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.