Methodologic Guide for Evaluating Clinical Performance and Effect of Artificial Intelligence Technology for Medical Diagnosis and Prediction
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
The use of artificial intelligence in medicine is currently an issue of great interest, especially with regard to the diagnostic or predictive analysis of medical images. Adoption of an artificial intelligence tool in clinical practice requires careful confirmation of its clinical utility. Herein, the authors explain key methodology points involved in a clinical evaluation of artificial intelligence technology for use in medicine, especially high-dimensional or overparameterized diagnostic or predictive models in which artificial deep neural networks are used, mainly from the standpoints of clinical epidemiology and biostatistics. First, statistical methods for assessing the discrimination and calibration performances of a diagnostic or predictive model are summarized. Next, the effects of disease manifestation spectrum and disease prevalence on the performance results are explained, followed by a discussion of the difference between evaluating the performance with use of internal and external datasets, the importance of using an adequate external dataset obtained from a well-defined clinical cohort to avoid overestimating the clinical performance as a result of overfitting in high-dimensional or overparameterized classification model and spectrum bias, and the essentials for achieving a more robust clinical evaluation. Finally, the authors review the role of clinical trials and observational outcome studies for ultimate clinical verification of diagnostic or predictive artificial intelligence tools through patient outcomes, beyond performance metrics, and how to design such studies. © RSNA, 2018
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This common scenario means that estimations of AI performance based on internal test data may overestimate its real-world performance on external data [73–75].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [74]
In the citing paper
“Similar to internal validation of the AIPM, external AIPM model validation can be based on discrimination (area under the receiver operating characteristic curve, sensitivity, specificity, positive and negative predictive values), calibration (calibration plot)5,6,13,17,49,56,78, and Decision Curve Analysis5,6,17,49,68,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“To ensure the generalizability of the AIPM to the intended healthcare setting, developers are advised to extensively validate the model for representative data from that setting6–8,10,13–15,24,27,56,64,66,68,72,77,79,87,88 (see phase 1, Representativeness).”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“The literature recommends to assess generalizability on external data from a different time period, place, or healthcare setting3,7,8,10,16,17,27,56,68,72,79.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“Although prospective validation is rare, it is preferred by the literature5,14,56, as it provides a better idea of the AIPM’s true applicability to medical practice and allows the healthcare professionals to identify and review errors in real time19,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
(2016) · Cited by 11,679
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293
Lakhani, P., & Sundaram, B. (2017) · Radiology · doi:10.1148/radiol.2017162326 · Cited by 1,775
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Radiology · doi:10.1148/radiol.2015151516 · Cited by 1,337
Deep Learning in Medical Imaging: General Overview
Lee, J.-G., Jun, S., Cho, Y.-W., Lee, H., Kim, G. B., Seo, J. B., & Kim, N. (2017) · Korean Journal of Radiology · doi:10.3348/kjr.2017.18.4.570 · Cited by 1,276
Research works printed but not linked: 3
- Steyerberg EW, 2010, Clinical prediction models: a practical approach to development, validation, and updating, 83
- Steyerberg EW, 2010, Clinical prediction models: a practical approach to development, validation, and updating, 299
- Newman TB, 2013, Designing clinical research, 117
Cited by
Items 101–200 of 1,000
Success Factors of Artificial Intelligence Implementation in Healthcare
Wolff, J., Pauling, J., Keck, A., & Baumbach, J. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.594971 · Cited by 104
“data governance, (4) transparency, (5) diversity, non-discrimination,”
BackgroundVoter, A. F., Meram, E., Garrett, J. W., & Yu, J. J. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.03.005 · Cited by 103
Lee, J. H., Ha, E. J., Kim, D., Jung, Y. J., Heo, S., Jang, Y.-H., An, S. H., & Lee, K. (2020) · European Radiology · doi:10.1007/s00330-019-06652-4 · Cited by 101
Introduction to Artificial Intelligence and Machine Learning for Pathology
Harrison, J. H., Gilbertson, J. R., Hanna, M. G., Olson, N. H., Seheult, J. N., Sorace, J. M., & Stram, M. N. (2021) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2020-0541-cp · Cited by 82
“128,131,132 Several metrics have also been used to express the adequacy of calibration, including the slope and intercept of the reliability diagram (Figure 7, G), and the overall observed versus predicted (observed/expected) ratio, which expresses average calibration across the operating range,…”
BackgroundMethodTechnical and clinical overview of deep learning in radiology
Ueda, D., Shimazaki, A., & Miki, Y. (2019) · Japanese Journal of Radiology · doi:10.1007/s11604-018-0795-3 · Cited by 100
Assessing the Economic Value of Clinical Artificial Intelligence: Challenges and Opportunities
Hendrix, N., Veenstra, D. L., Cheng, M., Anderson, N. C., & Verguet, S. (2022) · Value in Health · doi:10.1016/j.jval.2021.08.015 · Cited by 83
A pilot study of a deep learning approach to detect marginal bone loss around implants
Liu, M., Wang, S., Hu, C., & Liu, Y. (2022) · BMC Oral Health · doi:10.1186/s12903-021-02035-8 · Cited by 99
Jha, N., Lee, K.-S., & Kim, Y.-J. (2022) · PLOS ONE · doi:10.1371/journal.pone.0272715 · Cited by 64
Detection of cerebral aneurysms using artificial intelligence: a systematic review and meta-analysis
Din, M., Agarwal, S., Grzeda, M. T., Wood, D. A., Modat, M., & Booth, T. C. (2022) · Journal of NeuroInterventional Surgery · doi:10.1136/jnis-2022-019456 · Cited by 96
Nguyen, H. S., Ho, D. K. N., Nguyen, N. N., Tran, H. M., Tam, K., & Le, N. Q. K. (2023) · Academic Radiology · doi:10.1016/j.acra.2023.03.040 · Cited by 96
Ferrari, R., Mancini-Terracciano, C., Voena, C., Rengo, M., Zerunian, M., Ciardiello, A., Grasso, S., Marè, V., Paramatti, R., Russomando, A., et al. (2019) · European Journal of Radiology · doi:10.1016/j.ejrad.2019.06.013 · Cited by 94
Wichmann, J. L., Willemink, M. J., & De Cecco, C. N. (2020) · Investigative Radiology · doi:10.1097/rli.0000000000000673 · Cited by 79
“Park and Han 63 recently published a thorough guideline on methodological evaluation of clinical performance and impact of AI algorithms.”
Stib, M. T., Vasquez, J., Dong, M. P., Kim, Y. H., Subzwari, S. S., Triedman, H. J., Wang, A., Wang, H.-L. C., Yao, A. D., Jayaraman, M., et al. (2020) · Radiology · doi:10.1148/radiol.2020200334 · Cited by 84
“The strengths of our study include a varied patient population from multiple hospitals and CT scanners with variations in technical parameters (37).”
BackgroundPemberton, H., Zaki, L. A. M., Goodkin, O., Das, R., Steketee, R. M. E., Barkhof, F., & Vernooij, M. W. (2021) · Neuroradiology · doi:10.1007/s00234-021-02746-3 · Cited by 90
Shao, J., Ma, J., Zhang, Q., Li, W., & Wang, C. (2023) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2023.02.006 · Cited by 90
Prospects of deep learning for medical imaging
Kim, J., Hong, J., & Park, H. (2018) · Precision and Future Medicine · doi:10.23838/pfm.2018.00030 · Cited by 89
Park, S., Lee, S. M., Lee, K. H., Jung, K.-H., Bae, W., Choe, J., & Seo, J. B. (2019) · European Radiology · doi:10.1007/s00330-019-06532-x · Cited by 85
“Third, because our data were collected separately from disease-positive and diseasenegative subjects on the basis of radiologic reports in a casecontrolled manner, which is referred to as convenience sampling [22], spectrum bias may have been introduced.”
BackgroundSelf-citationAdvancements in Oncology with Artificial Intelligence—A Review Article
Vobugari, N., Raja, V., Sethi, U., Gandhi, K. D., Raja, K., & Surani, S. (2022) · Cancers · doi:10.3390/cancers14051349 · Cited by 85
Koçak, B. (2022) · Diagnostic and Interventional Radiology · doi:10.5152/dir.2022.211297 · Cited by 85
“19,20 Such distribution differences or shifts are highly likely to cause significant performance problems in future predictions.”
BackgroundUeda, D., Matsumoto, T., Ehara, S., Yamamoto, A., Walston, S. L., Ito, A., Shimono, T., Shiba, M., Takeshita, T., Fukuda, D., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00107-3 · Cited by 84
Artificial intelligence in thyroid ultrasound
Cao, C.-L., Li, Q., Jin, T., Shi, L., Li, W., Ya, X., Cheng, J., Du, T., Li, J., & Cui, X. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1060702 · Cited by 84
Basic of machine learning and deep learning in imaging for medical physicists
Manco, L., Maffei, N., Strolin, S., Vichi, S., Bottazzi, L., & Strigari, L. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.03.026 · Cited by 83
“The use of models based on training data which is not representative of the population, case mix, modalities, and acquisition protocols can compromise performance and confidence in its use, particularly if over-fitting has occurred [89].”
MethodChen, Q., Zhang, L., Mo, X., You, J., Chen, L., Jin, F., Wang, F., Jin, Z., Zhang, B., & Zhang, S. (2021) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-021-05509-7 · Cited by 82
“Typically, the internal validation, used for a preliminary evaluation or for the fine-tuning of the model under development, overestimates the performance [40].”
BackgroundInfluentialHuang, W., Chen, Y., Fedorov, A., Li, X., Jajamovich, G. H., Malyarenko, D. I., Aryal, M. P., LaViolette, P. S., Oborski, M. J., O’Sullivan, F., et al. (2019) · Tomography · doi:10.18383/j.tom.2018.00027 · Cited by 82
Derevianko, A., Pizzoli, S. F. M., Pesapane, F., Rotili, A., Monzani, D., Grasso, R., Cassano, E., & Pravettoni, G. (2023) · Cancers · doi:10.3390/cancers15020470 · Cited by 81
“In the last four decades, medical technology has seen a shift in the development of Artificial Intelligence (AI) which is commonly defined as “a field of computer science that develops systems able to perform tasks commonly associated with intelligent human beings” [1].”
BackgroundHwang, E. J., & Park, C. M. (2020) · Korean Journal of Radiology · doi:10.3348/kjr.2019.0821 · Cited by 71
How AI May Transform Musculoskeletal Imaging
Guermazi, A., Omoumi, P., Tordjman, M., Fritz, J. M., Kijowski, R., Regnard, N.-E., Carrino, J. A., Kahn, C. E., Knöll, F., Rueckert, D., et al. (2024) · Radiology · doi:10.1148/radiol.230764 · Cited by 80
Guidelines os the Brazilian Society of Cardiology on Telemedicine in Cardiology - 2019
Lopes, M. A. C. Q., de Oliveira, G. M. M., Ribeiro, A. L. P., Pinto, F. J., Rey, H. C. V., o, A. ́. A. ́. B., Zimerman, L. I., Hajjar, L. A., Rochitte, C. E., Bacal, F., et al. (2019) · Arquivos Brasileiros de Cardiologia · doi:10.5935/abc.20190205 · Cited by 80
Moon, S. J., Hwang, J., Kana, R. K., Torous, J. B., & Kim, J. W. (2019) · JMIR Mental Health · doi:10.2196/14108 · Cited by 80
“However, when a study provided no information about the validation process, low RoB was assumed if independent datasets were used for training, building a model, and validation [15].”
BackgroundYagi, M., Yamanouchi, K., Fujita, N., Funao, H., & Ebata, S. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12134188 · Cited by 79
“They need to be thoroughly evaluated and confirmed through rigorous testing, validation, and continuous assessment to ensure the accuracy and reliability of their outputs (Table 4) [52].”
BackgroundLarge Language Models in Medical Diagnostics: Scoping Review With Bibliometric Analysis
Su, H., Sun, Y., Li, R., Zhang, A., Yang, Y., Xiao, F., Duan, Z., Chen, J., Hu, Q., Yang, T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/72062 · Cited by 79
“To bolster these processes, artificial intelligence (AI) models have been increasingly used, demonstrating the potential to rival the diagnostic prowess of seasoned clinicians [1,2].”
Workflow Applications of Artificial Intelligence in Radiology and an Overview of Available Tools
Kapoor, N., Lacson, R. C., & Khorasani, R. (2020) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.08.016 · Cited by 78
Bayesian networks in healthcare: What is preventing their adoption?
Kyrimi, E., Dube, K., Fenton, N., Fahmi, A., Neves, M. R., Marsh, W., & McLachlan, S. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102079 · Cited by 81
Lee, M. K., Park, J. E., Jo, Y., Park, S. Y., Kim, S. J., & Kim, H. S. (2019) · European Radiology · doi:10.1007/s00330-019-06395-2 · Cited by 76
“The diagnostic performance of imaging parameters is not affected by disease prevalence, which was determined by AUC, sensitivity and specificity [28].”
MethodSelf-citationSuh, C. H., Shim, W. H., Kim, S. J., Roh, J. H., Lee, J., Kim, M.-J., Park, S., Jung, W., Sung, J., Jahng, G., et al. (2020) · American Journal of Neuroradiology · doi:10.3174/ajnr.a6848 · Cited by 76
“Evaluation of the clinical performance of a diagnostic or predictive artificial intelligence model requires the analysis of external data from a clinical cohort that appropriately represents the target patient population, avoiding overestimation of the initial results because of overfitting and spectrum bias.(32) Our…”
MethodSelf-citationSyer, T., Mehta, P. J., Antonelli, M., Mallett, S., Atkinson, D., Ourselin, S., & Punwani, S. (2021) · Cancers · doi:10.3390/cancers13133318 · Cited by 75
“Studies should determine the minimum sample size required to adequately power a study to detect some clinically relevant effect size for CAD and to allow statistically valid comparisons [48]; among the studies included in this review, only Gaur et al.”
BackgroundUrmán, J., Herranz, J. M., Uriarte, I., Rullán, M., Oyón, D., González, B., Fernández-Urién, I., Carrascosa, J., Bolado, F., Zabalza, L., et al. (2020) · Cancers · doi:10.3390/cancers12061644 · Cited by 75
Artificial Intelligence: A Primer for Breast Imaging Radiologists
Bahl, M. (2020) · Journal of Breast Imaging · doi:10.1093/jbi/wbaa033 · Cited by 74
“Model performance should also be validated with a completely external dataset (eg, data collected by independent investigators at a different site) to confirm its generalizability for clinical practice (42).”
Deep learning applied to automatic disease detection using chest X‐rays
Moses, D. A. (2021) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13273 · Cited by 74
“6 If we analyse localisation performance where we have multiple regions of disease (e.g. lung nodules) in the same CXR, then the free-response operating characteristic (FROC) is useful.”
MethodMachine Learning in the Diagnosis and Prognostic Prediction of Dental Caries: A Systematic Review
Reyes, L. T., Knorst, J. K., Ortiz, F. R., & Ardenghi, T. M. (2022) · Caries Research · doi:10.1159/000524167 · Cited by 73
“Consensus must also be established for the preprocessing and labeling of the data [Rajkomar et al., 2019; Park and Han, 2018].”
BackgroundMethodInfluentialArtificial intelligence from A to Z: From neural network to legal framework
van Assen, M., Lee, S. J., & De Cecco, C. N. (2020) · European Journal of Radiology · doi:10.1016/j.ejrad.2020.109083 · Cited by 62
Ninatti, G., Kirienko, M., Neri, E., Sollini, M., & Chiti, A. (2020) · Diagnostics · doi:10.3390/diagnostics10060359 · Cited by 71
“Benefitting from technical variability aspects, it measures better the model’s performance and provides a proof of generalizability [52,83].”
BackgroundAnderson, A. W., Marinovich, M. L., Houssami, N., Lowry, K. P., Elmore, J. G., Buist, D. S. M., Hofvind, S. S. H., & Lee, C. I. (2022) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.11.008 · Cited by 74
Park, S. H., & Kressel, H. Y. (2018) · Journal of Korean Medical Science · doi:10.3346/jkms.2018.33.e152 · Cited by 66
Self-citationPark, S., Lee, S. M., Kim, N., Choe, J., Cho, Y., Do, K.-H., & Seo, J. B. (2019) · European Radiology · doi:10.1007/s00330-019-06130-x · Cited by 71
Self-citationDeep learning in spatiotemporal cardiac imaging: A review of methodologies and clinical usability
Hernandez, K. A. L., Rienmüller, T. M., Baumgartner, D., & Baumgärtner, C. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104200 · Cited by 71
Fold-stratified cross-validation for unbiased and privacy-preserving federated learning
Bey, R., Goussault, R., Grolleau, F., Benchoufi, M., & Porcher, R. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa096 · Cited by 70
“In addition to privacy concerns, recent controversies indicate that many AI models may have been validated improperly, shedding doubt on the performances that have been advertized [31–35].”
BackgroundOne Algorithm May Not Fit All: How Selection Bias Affects Machine Learning Performance
Yu, A. C., & Eng, J. (2020) · RadioGraphics · doi:10.1148/rg.2020200040 · Cited by 59
Machine learning based natural language processing of radiology reports in orthopaedic trauma
Olthof, A. W., Shouche, P., Fennema, E. M., IJpma, F. F. A., Koolstra, R. H. C., Stirler, V. M. A., van Ooijen, P. M. A., & Cornelissen, L. J. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106304 · Cited by 70
“Temporal and external validation is required before clinical application [ 40 , 41 ].”
BackgroundSELF: a stacked-based ensemble learning framework for breast cancer classification
Jakhar, A. K., Gupta, A., & Singh, M. (2023) · Evolutionary Intelligence · doi:10.1007/s12065-023-00824-4 · Cited by 70
“The proper and effective disease treatment with certain information is demanding and challenging in the area of bioinformatics and medical research [33].”
Villacampa, G. J., Tung, N., Pernas, S., Paré, L., Bueno, C., Echavarría, I., López‐Tarruella, S., Roche-Molina, M., del Monte-Millán, M., Marín‐Aguilera, M., et al. (2023) · Annals of Oncology · doi:10.1016/j.annonc.2023.05.012 · Cited by 70
Kim, H., Lee, D., Cho, W. S., Lee, J. C., Goo, J. M., Kim, H. C., & Park, C. M. (2020) · European Radiology · doi:10.1007/s00330-019-06628-4 · Cited by 69
Real-World Performance of Computer-Aided Diagnosis System for Thyroid Nodules Using Ultrasonography
Kim, H. L., Ha, E. J., & Han, M. (2019) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2019.05.032 · Cited by 69
HyVADRF: Hybrid VADER–Random Forest and GWO for Bitcoin Tweet Sentiment Analysis
Mardjo, A., & Choksuchat, C. (2022) · IEEE Access · doi:10.1109/access.2022.3209662 · Cited by 68
“174 In a recent study [31], the importance of obtaining adequately 175 sized, unbiased validation and training sets was identified as 176 a crucial factor in the assessment and development of robust 177 machine learning models.”
BackgroundQu, W., Balki, I., Mendez, M., Valen, J., Levman, J. E. D., & Tyrrell, P. N. (2020) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-020-02260-6 · Cited by 67
“In practice, the most informative result for patients is the probability of having a disease, which depends on both disease prevalence in the population that the patient belongs to (pretest probability) and the model prediction [13].”
BackgroundMethodLehnen, N. C., Haase, R., Faber, J., Rüber, T., Vatter, H., Radbruch, A., & Schmeel, F. C. (2021) · Diagnostics · doi:10.3390/diagnostics11050902 · Cited by 67
“External validation is an important step in the validation of a deep-learning-based, predictive model to avoid overfitting to the training data set, potentially resulting in an overestimation of the model’s diagnostic performance [38,39].”
Hung, K. F., Ai, Q. Y. H., King, A. D., Bornstein, M. M., Wong, L. M., & Leung, Y. Y. (2022) · Clinical Oral Investigations · doi:10.1007/s00784-021-04365-x · Cited by 67
“The training-monitoring dataset unseen by the network was used to monitor the algorithm for overfitting during the course of training [24].”
MethodMiao, Z. (2019) · The Electronic Library · doi:10.1108/el-04-2019-0089 · Cited by 69
Fundamentals in Artificial Intelligence for Vascular Surgeons
Raffort, J., Adam, C., Carrier, M., & Lareyre, F. (2019) · Annals of Vascular Surgery · doi:10.1016/j.avsg.2019.11.037 · Cited by 67
Choi, J. W., Cho, Y. J., Ha, J. Y., Lee, S. B., Lee, S., Choi, Y. H., Cheon, J., & Kim, W. S. (2021) · Scientific Reports · doi:10.1038/s41598-021-00058-3 · Cited by 66
Yeo, M., Tahayori, B., Kok, H. K., Maingard, J., Kutaiba, N., Russell, J., Thijs, V., Jhamb, A., Chandra, R. V., Brooks, M., et al. (2021) · Journal of NeuroInterventional Surgery · doi:10.1136/neurintsurg-2020-017099 · Cited by 65
Decision tree and artificial immune systems for stroke prediction in imbalanced data
Santos, L. I., Camargos, M. O., D’Ângelo, M. F. S. V., Mendes, J. B., de Medeiros, E. E. C., Guimarães, A. L. S., & Palhares, R. M. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.116221 · Cited by 66
Matsoukas, S., Scaggiante, J., Schuldt, B. R., Smith, C. J., Chennareddy, S., Kalagara, R., Majidi, S., Bederson, J. B., Fifi, J. T., Mocco, J., et al. (2022) · La radiologia medica · doi:10.1007/s11547-022-01530-4 · Cited by 65
“Finally, validation refers to the “fine-tuning’’ stage of the developing model which takes part in the final stages of training, usually in smaller datasets that focus on the exact patient population of the test [4].”
BackgroundLee, J. H., Kim, K. H., Lee, E. H., Ahn, J. S., Ryu, J. K., Park, Y. M., Shin, G. W., Kim, Y. J., & Choi, H. Y. (2022) · Korean Journal of Radiology · doi:10.3348/kjr.2021.0476 · Cited by 63
“Although AI algorithms show great potential in breast cancer screening, their characteristics must be carefully evaluated and radiologists should fully understand these characteristics when they are used in practice [20,21].”
Taylor‐Phillips, S., Seedat, F., Kijauskaite, G., Marshall, J., Halligan, S., Hyde, C., Given-Wilson, R., Wilkinson, L., Denniston, A. K., Glocker, B., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00088-7 · Cited by 62
“34,35 The least biased study designs for measuring impact on women's outcomes are prospective, testtreat, randomised controlled trials that randomly allocate women to the original pathway (two radiologists plus arbitration) or the new proposed pathway, with followup of women to clinically…”
Afolayan, J. O., Adebiyi, M. O., Arowolo, M. O., Chakraborty, C., & Adebiyi, A. A. (2022) · Springer eBooks · doi:10.1007/978-981-16-8150-9_4 · Cited by 62
Jo, H., Kim, C., Gwon, D., Lee, J. H., Lee, J., Park, K. M., & Park, S.-H. (2023) · Scientific Reports · doi:10.1038/s41598-023-44201-8 · Cited by 62
“Next, we plotted a calibration plot and calculated the Brier score, which represents the mean squared difference between the predicted probability and the true outcome, to evaluate the models’ calibration performances 23,24 .”
MethodSelf-citationPark, J. E., Kim, H. S., Park, S. Y., Nam, S. J., Chun, S., Jo, Y., & Kim, J. H. (2019) · Radiology · doi:10.1148/radiol.2019190913 · Cited by 62
Self-citationOutlining the Design Space of Explainable Intelligent Systems for Medical DiagnosisPreprint
Xie, Y., Gao, G., & Chen, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06019 · Cited by 62
“Another research proposed the methods of evaluating the clinical performance and effect of the artificial intelligent systems in medical diagnosis and one of the methods mentioned the explanations [39].”
MethodAn artificial intelligence-based risk prediction model of myocardial infarction
Liu, R., Wang, M., Zheng, T., Zhang, R., Li, N., Chen, Z., Yan, H., & Shi, Q. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04761-4 · Cited by 61
“Artificial intelligence (AI) is a technology of high research interest that can be applied to clinical diagnosis [7, 8].”
BackgroundConvolutional Neural Networks for classifying skin lesions
Pai, K., & Giridharan, A. (2019) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon.2019.8929461 · Cited by 59
Artificial Intelligence in Coronary Computed Tomography Angiography: From Anatomy to Prognosis
Muscogiuri, G., van Assen, M., Tesche, C., Cecco, C. N. D., Chiesa, M., Scafuri, S., Guglielmo, M., Baggiano, A., Fusini, L., Guaricci, A. I., et al. (2020) · BioMed Research International · doi:10.1155/2020/6649410 · Cited by 59
“Since then, several guidelines have been published to improve the validation process of medical AI applications [23, 24].”
BackgroundAlis, D. C., Aliş, C., Yergin, M., Topel, Ç., Asmakutlu, O., Bağcılar, Ö., Senli, Y. D., Üstündağ, A., Salt, V., Doğan, S. N., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-05872-x · Cited by 59
“Nevertheless, it is well-known that DL models’ performance should be explored on unseen test data, preferentially on an external sample, to precisely uncover the models’ generalizability 15 .”
BackgroundHanna, M. G., Olson, N. H., Zarella, M. D., Dash, R. C., Herrmann, M. D., Furtado, L. V., Stram, M. N., Raciti, P. M., Hassell, L. A., Mays, A., et al. (2023) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2023-0042-cp · Cited by 58
Applications of Artificial Intelligence in Breast Imaging
Morgan, M. B., & Mates, J. (2020) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2020.08.007 · Cited by 59
Park, S. H. (2018) · Radiology · doi:10.1148/radiol.2018182294 · Cited by 58
Self-citationArtificial intelligence for dementia research methods optimization
Bucholc, M., James, C., Khleifat, A. A., Badhwar, A., Clarke, N., Dehsarvi, A., Madan, C. R., Marzi, S. J., Shand, C., Schilder, B. M., et al. (2023) · Alzheimer's & Dementia · doi:10.1002/alz.13441 · Cited by 40
Banzato, T., Causin, F., Puppa, A. D., Cester, G., Mazzai, L., & Zotti, A. (2019) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.26723 · Cited by 57
Applications of machine learning and deep learning to thyroid imaging: where do we stand?
Ha, E. J., & Baek, J. H. (2021) · Ultrasonography · doi:10.14366/usg.20068 · Cited by 47
“However, at this point, the use of AI tools in clinical practice is of great concern since most studies were designed as proof-ofconcept or technical feasibility research without a thorough external validation of real-world clinical performance [14-16].”
BackgroundWong, L. M., King, A. D., Ai, Q. Y. H., Lam, W. K., Poon, D. M. C., Ma, B. B. Y., Chan, K. C. A., & Mo, F. K. F. (2020) · European Radiology · doi:10.1007/s00330-020-07451-y · Cited by 57
Reyes, L. T., Knorst, J. K., Ortiz, F. R., & Ardenghi, T. M. (2021) · Journal of Clinical and Translational Research · doi:10.18053/jctres.07.202104.012 · Cited by 56
“Emphasis should be placed on the availability of data from real examples for subsequent application of the model in clinical practice [39,106].”
BackgroundMethodInfluentialHong, G.-S., Jang, M., Kyung, S., Cho, K., Jeong, J., Lee, G. Y., Shin, K., Kim, K., Ryu, S. M., Seo, J. B., et al. (2023) · Korean Journal of Radiology · doi:10.3348/kjr.2023.0393 · Cited by 57
“…Library [160,161] A recent article pointed out that most research on the use of AI in medical imaging lacks standardized study designs appropriate for real-world clinical settings [162], whereas another study put forth relevant methods for evaluating the clinical performance of medical DL [163].”
MethodDeep learning performance for detection and classification of microcalcifications on mammography
Pesapane, F., Trentin, C., Ferrari, F., Signorelli, G., Tantrige, P. M., Montesano, M., Cicala, C., Virgoli, R., D'Acquisto, S., Nicosia, L., et al. (2023) · European Radiology Experimental · doi:10.1186/s41747-023-00384-3 · Cited by 57
“Overall, our data was heterogeneous and representative of the population, structured, annotated, and ready to use, which is something limited, currently existing in only a small number of institutions [35].”
BackgroundMethodArtificial intelligence in breast cancer screening: primary care provider preferences
Hendrix, N., Hauber, B., Lee, C. I., Bansal, A., & Veenstra, D. L. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa292 · Cited by 56
“33 We also identified meaningful heterogeneity in PCPs’ preferences, which we explained by using a latent class model to group respondents into 3 classes.”
MethodCarmo, L. O. E., van den Merkhof, A., Olczak, J., Gordon, M., Jutte, P. C., Jaarsma, R. L., IJpma, F. F. A., Doornberg, J. N., & Prijs, J. (2021) · Bone & Joint Open · doi:10.1302/2633-1462.210.bjo-2021-0133 · Cited by 56
Martín, M., Stecklein, S. R., Gluz, O., Villacampa, G. J., del Monte-Millán, M., Nitz, U. A., Cobo, S., Christgen, M., Brasó‐Maristany, F., Álvarez, E. L., et al. (2024) · Annals of Oncology · doi:10.1016/j.annonc.2024.10.012 · Cited by 55
“29 The second objective to validate the model was to assess the ability of the TNBC-DX risk score (as a continuous variable and group categories) to predict survival outcome (DDFS and OS) in the independent validation cohorts without pembrolizumab.”
Radiomics and Deep Learning from Research to Clinical Workflow: Neuro-Oncologic Imaging
Park, J. E., Kickingereder, P., & Kim, H. S. (2020) · Korean Journal of Radiology · doi:10.3348/kjr.2019.0847 · Cited by 55
“A prospective diagnostic cohort design is a way to adequately represent the manifestation spectrum of target patients in real-world clinical settings, with a before-after study design being applied (27).”
MethodArtificial intelligence in breast ultrasonography
Kim, J., Kim, H. J., Kim, C., & Kim, W. H. (2021) · Ultrasonography · doi:10.14366/usg.20117 · Cited by 53
“Further studies or guidelines may be necessary to specify the structure and details of the dataset according to the type of dataset (training/validation/test) for a generalizable AI system, by minimizing selection and spectrum bias [17,18].”
BackgroundChristou, C. D., & Tsoulfas, G. (2021) · World Journal of Gastroenterology · doi:10.3748/wjg.v27.i37.6191 · Cited by 54
Disease analysis using machine learning approaches in healthcare system
Bhuyan, H. K., Ravi, V., Brahma, B., & Kamila, N. K. (2022) · Health and Technology · doi:10.1007/s12553-022-00687-2 · Cited by 54
“Precise diagnosis issues are the major problem in bioinformatics or medical investigation [1].”
BackgroundArtificial intelligence: A review of current applications in hepatocellular carcinoma imaging
Pellat, A., Barat, M., Coriat, R., Soyer, P. A., & Dohan, A. J. J. (2022) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2022.10.001 · Cited by 54
Artificial Intelligence–assisted Prostate Cancer Diagnosis: Radiologic-Pathologic Correlation
Mata, L. A., Retámero, J. A., Gupta, R. T., Figueras, R. G., & Luna, A. (2021) · Radiographics · doi:10.1148/rg.2021210020 · Cited by 53
Artificial Intelligence in Lymphoma PET Imaging
Hasani, N., Paravastu, S. S., Farhadi, F., Yousefirizi, F., Morris, M. A., Rahmim, A., Roschewski, M. J., Summers, R. M., & Saboury, B. (2021) · PET Clinics · doi:10.1016/j.cpet.2021.09.006 · Cited by 54
Mapping the Bibliometrics Landscape of AI in Medicine: Methodological Study
Shi, J., Bendig, D., Vollmar, H. C., & Rasche, P. (2023) · Journal of Medical Internet Research · doi:10.2196/45815 · Cited by 53
“Its potential for revolutionizing medical practices, enhancing patient care quality, and improving health care efficiency has spurred substantial research endeavors [10-14], with recent years witnessing an unprecedented surge in medical AI studies [3,15,16].”
Jin, J., Jiang, Y., Zhao, Y., & Huang, P. (2023) · Academic Radiology · doi:10.1016/j.acra.2023.09.008 · Cited by 53
Haenssle, H. A., Winkler, J. K., Fink, C., Toberer, F., Enk, A., Stolz, W., Deinlein, T., Hofmann‐Wellenhof, R., Kittler, H., Tschandl, P., et al. (2020) · European Journal of Cancer · doi:10.1016/j.ejca.2020.11.034 · Cited by 53
Thian, Y. L., Ng, D., Hallinan, J. T. P. D., Jagmohan, P., Sia, S. Y., Tan, C. H., Ting, Y. H., Kei, P. L., Pulickal, G. G., Tiong, V. T. Y., et al. (2021) · Radiology Artificial Intelligence · doi:10.1148/ryai.2021200190 · Cited by 53
“In clinical practice, the operating point for sensitivity and specificity estimates is directed by disease prevalence (27), and the best threshold to use depends on the trade-offs among the positive predictive value, negative predictive value, sensitivity, and specificity (28).”
BackgroundRajpurkar, P., Joshi, A., Pareek, A., Chen, P., Kiani, A., Irvin, J., Ng, A. Y., & Lungren, M. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11379 · Cited by 55
“Furthermore, the results of our study do not suggest guaranteed generalization of chest x-ray models to new clinical settings; future work should evaluate evaluate the performance in clinical trials for further verification, a necessary step for the successful translation of diagnostic or predictive artificial intelli…”
BackgroundNeisius, U., El-Rewaidy, H., Küçükseymen, S., Tsao, C. W., Mâncio, J., Nakamori, S., Manning, W. J., & Nezafat, R. (2020) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.27048 · Cited by 52
“3).(31) To balance the data between LGE(+) and LGE(−) slices during training, 50% of LGE(−) slices were randomly discarded, resulting in the training dataset (T1 maps, n = 377).”
MethodCabitza, F., Campagner, A., Angius, R., Natali, C., & Reverberi, C. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581095 · Cited by 51
“Our contribution marks a discontinuity with that evaluative approach, concerned as it is in diferential impact, rather than improbable performance scores [57, 76] , that is, in the efect in terms of either improvement and degradation of a decision-making team once its members use a decision support (of comparable or s…”
Background
Metadata
- Authors
- Park, S. H., & Han, K.
- Affiliations
- Yonsei University (KR)
- Venue
- Radiology
- Published
- 2018-03
- Volume
- 286
- Issue
- 3
- Pages
- 800-809
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2783687327
- Cited by
- 894
- References
- 49
- Citations by year
- 2026: 123 · 2025: 98 · 2024: 96 · 2023: 135 · 2022: 111 · 2021: 121 · 2020: 103 · 2019: 80 · 2018: 25
- FWCI
- 1.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Ministry of Trade, Industry and Energy
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- artificial intelligence in healthcare, deep neural network, medical image diagnosis, prediction model, discrimination and calibration, overfitting, spectrum bias, external validation, disease prevalence, outcomes research, high-dimensional models
- MeSH terms
- Artificial Intelligence, Data Interpretation, Statistical, Diagnostic Imaging, Humans, Image Interpretation, Computer-Assisted, Terminology as Topic, Predictive Value of Tests, Technology Assessment, Biomedical, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.