Methodologic Guide for Evaluating Clinical Performance and Effect of Artificial Intelligence Technology for Medical Diagnosis and Prediction
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
The use of artificial intelligence in medicine is currently an issue of great interest, especially with regard to the diagnostic or predictive analysis of medical images. Adoption of an artificial intelligence tool in clinical practice requires careful confirmation of its clinical utility. Herein, the authors explain key methodology points involved in a clinical evaluation of artificial intelligence technology for use in medicine, especially high-dimensional or overparameterized diagnostic or predictive models in which artificial deep neural networks are used, mainly from the standpoints of clinical epidemiology and biostatistics. First, statistical methods for assessing the discrimination and calibration performances of a diagnostic or predictive model are summarized. Next, the effects of disease manifestation spectrum and disease prevalence on the performance results are explained, followed by a discussion of the difference between evaluating the performance with use of internal and external datasets, the importance of using an adequate external dataset obtained from a well-defined clinical cohort to avoid overestimating the clinical performance as a result of overfitting in high-dimensional or overparameterized classification model and spectrum bias, and the essentials for achieving a more robust clinical evaluation. Finally, the authors review the role of clinical trials and observational outcome studies for ultimate clinical verification of diagnostic or predictive artificial intelligence tools through patient outcomes, beyond performance metrics, and how to design such studies. © RSNA, 2018
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This common scenario means that estimations of AI performance based on internal test data may overestimate its real-world performance on external data [73–75].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [74]
In the citing paper
“Similar to internal validation of the AIPM, external AIPM model validation can be based on discrimination (area under the receiver operating characteristic curve, sensitivity, specificity, positive and negative predictive values), calibration (calibration plot)5,6,13,17,49,56,78, and Decision Curve Analysis5,6,17,49,68,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“To ensure the generalizability of the AIPM to the intended healthcare setting, developers are advised to extensively validate the model for representative data from that setting6–8,10,13–15,24,27,56,64,66,68,72,77,79,87,88 (see phase 1, Representativeness).”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“The literature recommends to assess generalizability on external data from a different time period, place, or healthcare setting3,7,8,10,16,17,27,56,68,72,79.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
In the citing paper
“Although prospective validation is rare, it is preferred by the literature5,14,56, as it provides a better idea of the AIPM’s true applicability to medical practice and allows the healthcare professionals to identify and review errors in real time19,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [56]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
(2016) · Cited by 11,679
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293
Lakhani, P., & Sundaram, B. (2017) · Radiology · doi:10.1148/radiol.2017162326 · Cited by 1,775
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Radiology · doi:10.1148/radiol.2015151516 · Cited by 1,337
Deep Learning in Medical Imaging: General Overview
Lee, J.-G., Jun, S., Cho, Y.-W., Lee, H., Kim, G. B., Seo, J. B., & Kim, N. (2017) · Korean Journal of Radiology · doi:10.3348/kjr.2017.18.4.570 · Cited by 1,276
Machine Learning and Prediction in Medicine — Beyond the Peak of Inflated Expectations
Chen, J. H., & Asch, S. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1702071 · Cited by 1,171
Assessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066
(1975) · New England Journal of Medicine · doi:10.1056/nejm197507312930513 · Cited by 1,017
Yu, K.-H., Zhang, C., Berry, G. J., Altman, R. B., Ré, C., Rubin, D. L., & Snyder, M. (2016) · Nature Communications · doi:10.1038/ncomms12474 · Cited by 972
Receiver Operating Characteristic (ROC) Curve: Practical Review for Radiologists
Park, S. H., Goo, J. M., & Jo, C.-H. (2004) · Korean Journal of Radiology · doi:10.3348/kjr.2004.5.1.11 · Cited by 907
Self-citationReceiver Operating Characteristic Curves and Their Use in Radiology
Obuchowski, N. A. (2003) · Radiology · doi:10.1148/radiol.2291010898 · Cited by 843
Haukoos, J. S., & Lewis, R. J. (2015) · JAMA · doi:10.1001/jama.2015.13480 · Cited by 841
Sample Size Estimation: How Many Individuals Should Be Studied?
Eng, J. (2003) · Radiology · doi:10.1148/radiol.2272012051 · Cited by 811
Minimal Clinically Important Difference: Defining What Really Matters to Patients
McGlothlin, A. E., & Lewis, R. J. (2014) · JAMA · doi:10.1001/jama.2014.13128 · Cited by 703
The Potential of Radiomic-Based Phenotyping in Precision Medicine: A Review
Aerts, H. J. W. L. (2016) · JAMA Oncology · doi:10.1001/jamaoncol.2016.2631 · Cited by 669
Kirby, R. S. (2014) · Annals of Epidemiology · doi:10.1016/j.annepidem.2014.01.010 · Cited by 664
Clinical Prediction Models—a Practical Approach to Development, Validation and Updating
Bedogni, G. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2009.00614_16.x · Cited by 581
Confounding by Indication in Clinical Research
Kyriacou, D. N., & Lewis, R. J. (2016) · JAMA · doi:10.1001/jama.2016.16435 · Cited by 565
Evaluating Discrimination of Risk Prediction Models: The C Statistic
Pencina, M. J., & D’Agostino, R. B. (2015) · JAMA · doi:10.1001/jama.2015.11082 · Cited by 386
Sica, G. T. (2006) · Radiology · doi:10.1148/radiol.2383041109 · Cited by 331
Obuchowski (2005) · doi:10.2214/ajr.184.2.01840364 · Cited by 321
Implementing Machine Learning in Radiology Practice and Research
Kohli, M., Prevedello, L. M., Filice, R. W., & Geis, J. R. (2017) · American Journal of Roentgenology · doi:10.2214/ajr.16.17224 · Cited by 301
Prevedello, L. M., Erdal, B. S., Ryu, J. L., Little, K. J., Demirer, M., Qian, S., & White, R. D. (2017) · Radiology · doi:10.1148/radiol.2017162664 · Cited by 246
Sample size calculations in studies of test accuracy
Obuchowski, N. A. (1998) · Statistical Methods in Medical Research · doi:10.1177/096228029800700405 · Cited by 221
Relevance of deep learning to facilitate the diagnosis of HER2 status in breast cancer
Vandenberghe, M. E., Scott, M. L. J., Scorer, P. W., Söderberg, M., Balcerzak, D., & Barker, C. (2017) · Scientific Reports · doi:10.1038/srep45938 · Cited by 209
Brocklehurst, P., Field, D., Greene, K., Juszczak, E., Keith, R., Kenyon, S., Linsell, L., Mabey, C., Newburn, M., Plachcinski, R., et al. (2017) · The Lancet · doi:10.1016/s0140-6736(17)30568-8 · Cited by 207
Propensity Score Matching: A Conceptual Review for Radiology Researchers
Baek, S., Park, S. H., Won, E., Park, Y. R., & Kim, H. J. (2015) · Korean Journal of Radiology · doi:10.3348/kjr.2015.16.2.286 · Cited by 204
Self-citationHan, K., Song, K., & Choi, B. W. (2016) · Korean Journal of Radiology · doi:10.3348/kjr.2016.17.3.339 · Cited by 162
Self-citationAhn, S., Park, S. H., & Lee, K. H. (2013) · Radiology · doi:10.1148/radiol.12120725 · Cited by 159
Self-citationDeep Learning for Medical Image Analysis
Şenaras, Ç., & Gürcan, M. N. (2018) · Journal of Pathology Informatics · doi:10.4103/jpi.jpi_27_18 · Cited by 147
The two-step Fagan's nomogram: ad hoc interpretation of a diagnostic test result without calculation
Caraguel, C., & Vanderstichel, R. V. (2013) · BMJ Evidence-Based Medicine · doi:10.1136/eb-2013-101243 · Cited by 142
Uncontrolled before-after studies: discouraged by Cochrane and theEMJ
Goodacre, S. W. (2015) · Emergency Medicine Journal · doi:10.1136/emermed-2015-204761 · Cited by 136
The evidence base of clinical diagnosis : theory and methods of diagnostic research
Knottnerus, J. A., & Buntinx, F. (2009) · Cited by 119
The role of before-after studies of therapeutic impact in the evaluation of diagnostic technologies.
Guyatt, G., Tugwell, P., Feeny, D. H., Drummond, M., & Haynes, R. (1986) · Journal of Chronic Diseases · doi:10.1016/0021-9681(86)90051-2 · Cited by 91
Behind the Numbers: Propensity Score Analysis—A Primer for the Diagnostic Radiologist
McDonald, R. J., McDonald, J. S., Kallmes, D. F., & Carter, R. E. (2013) · Radiology · doi:10.1148/radiol.13131465 · Cited by 67
Brinkman, S. A., Johnson, S. E., Codde, J., Hart, M. B., Straton, J. A. Y., Mittinty, M., & Silburn, S. (2016) · The Lancet · doi:10.1016/s0140-6736(16)30384-1 · Cited by 65
on behalf of The INFANT Collaborative Group, & Brocklehurst, P. (2016) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-015-0780-0 · Cited by 46
Behind the Numbers: Inverse Probability Weighting
Halpern, E. F. (2014) · Radiology · doi:10.1148/radiol.14140035 · Cited by 43
Facilitating Prospective Registration of Diagnostic Accuracy Studies: A STARD Initiative
Korevaar, D. A., Hooft, L., Askie, L. M., Barbour, V., Faure, H., Gatsonis, C., Hunter, K. E., Kressel, H. Y., Lippman, H., McInnes, M. D. F., et al. (2017) · Clinical Chemistry · doi:10.1373/clinchem.2017.272765 · Cited by 36
Computerised cardiotocography—study design hampers findings
Belfort, M. A., & Clark, S. L. (2017) · The Lancet · doi:10.1016/s0140-6736(17)30762-6 · Cited by 12
Can additional experienced staff reduce emergency medical admissions?
Goodacre, S. W., Mason, S., Kersh, R., Webster, A., Samaniego, N., & Morris, F. (2004) · Emergency Medicine Journal · doi:10.1136/emj.2003.005876 · Cited by 11
Hosmer (1997) · doi:10.1002/(sici)1097-0258(19970515)16:9<965::aid-sim509>3.0.co;2-o
(1950) · doi:10.1002/1097-0142(1950)3:1<32::aid-cncr2820030106>3.0.co;2-3
Research works printed but not linked: 3
- Steyerberg EW, 2010, Clinical prediction models: a practical approach to development, validation, and updating, 83
- Steyerberg EW, 2010, Clinical prediction models: a practical approach to development, validation, and updating, 299
- Newman TB, 2013, Designing clinical research, 117
Cited by
Convolutional neural networks: an overview and application in radiology
Yamashita, R., Nishio, M., Do, R. K. G., & Togashi, K. (2018) · Insights into Imaging · doi:10.1007/s13244-018-0639-9 · Cited by 4,682
“It is worthy of mention that the term Bvalidation^ is used differently in the medical field and the machine learning field [26].”
BackgroundChecklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443
Preparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020
“This is especially true in the clinical setting and is well outlined by Park and Han (22).”
BackgroundArtificial intelligence for mental health and mental illnesses: an overview
Graham, S., Depp, C., Lee, E. E., Nebeker, C., Tu, X., Kim, H.-C., & Jeste, D. V. (2019) · Current Psychiatry Reports · doi:10.1007/s11920-019-1094-0 · Cited by 970
Lekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957
“Initial efforts have focused on providing recommendations for the reporting of AI studies for different medical domains or clinical tasks (eg, TRIPOD+AI, 11 CLAIM, 12 CONSORT-AI, 13 DECIDE-AI, 14 PROBAST-AI, 15 CLEAR 16 ).”
Metadata
- Authors
- Park, S. H., & Han, K.
- Affiliations
- Yonsei University (KR)
- Venue
- Radiology
- Published
- 2018-03
- Volume
- 286
- Issue
- 3
- Pages
- 800-809
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2783687327
- Cited by
- 894
- References
- 49
- Citations by year
- 2026: 123 · 2025: 98 · 2024: 96 · 2023: 135 · 2022: 111 · 2021: 121 · 2020: 103 · 2019: 80 · 2018: 25
- FWCI
- 1.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Ministry of Trade, Industry and Energy
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- artificial intelligence in healthcare, deep neural network, medical image diagnosis, prediction model, discrimination and calibration, overfitting, spectrum bias, external validation, disease prevalence, outcomes research, high-dimensional models
- MeSH terms
- Artificial Intelligence, Data Interpretation, Statistical, Diagnostic Imaging, Humans, Image Interpretation, Computer-Assisted, Terminology as Topic, Predictive Value of Tests, Technology Assessment, Biomedical, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.