The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Peering Into the Black Box of Artificial Intelligence: Evaluation Metrics of Machine Learning Methods
Open accessAbstract
OBJECTIVE: Machine learning (ML) and artificial intelligence (AI) are rapidly becoming the most talked about and controversial topics in radiology and medicine. Over the past few years, the numbers of ML- or AI-focused studies in the literature have increased almost exponentially, and ML has become a hot topic at academic and industry conferences. However, despite the increased awareness of ML as a tool, many medical professionals have a poor understanding of how ML works and how to critically appraise studies and tools that are presented to us. Thus, we present a brief overview of ML, explain the metrics used in ML and how to interpret them, and explain some of the technical jargon associated with the field so that readers with a medical background and basic knowledge of statistics can feel more comfortable when examining ML applications. CONCLUSION: Attention to sample size, overfitting, underfitting, cross validation, as well as a broad knowledge of the metrics of machine learning, can help those with little or no technical knowledge begin to assess machine learning studies. However, transparency in methods and sharing of algorithms is vital to allow clinicians to assess these tools themselves.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
An Introduction to Statistical Learning
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) · Springer Texts in Statistics · doi:10.1007/978-1-4614-7138-7 · Cited by 12,106
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
A systematic analysis of performance measures for classification tasks
Sokolova, M., & Lapalme, G. (2009) · Information Processing & Management · doi:10.1016/j.ipm.2009.03.002 · Cited by 7,515
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340
A survey of cross-validation procedures for model selection
Arlot, S., & Célisse, A. (2010) · LillOA (Université de Lille (University Of Lille)) · Cited by 3,669
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522
Deo, R. C. (2015) · Circulation · doi:10.1161/circulationaha.115.001593 · Cited by 3,466
Encyclopedia of Machine Learning
(2010) · Springer eBooks · doi:10.1007/978-0-387-30164-8 · Cited by 3,465
The use of receiver operating characteristic curves in biomedical informatics
Lasko, T. A., Bhagwat, J. G., Zou, K. H., & Ohno-Machado, L. (2005) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.02.008 · Cited by 823
Machine learning and radiology
Wang, S., & Summers, R. M. (2012) · Medical Image Analysis · doi:10.1016/j.media.2012.02.005 · Cited by 697
The coming of age of artificial intelligence in medicine
Patel, V. L., Shortliffe, E. H., Stefanelli, M., Szolovits, P., Berthold, M. R., Bellazzi, R., & Abu-Hanna, A. (2009) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2008.07.017 · Cited by 625
Machine Learning for Detection and Diagnosis of Disease
Sajda, P. (2006) · Annual Review of Biomedical Engineering · doi:10.1146/annurev.bioeng.8.061505.095802 · Cited by 409
Lever, J., Krzywinski, M., & Altman, N. (2016) · Nature Methods · doi:10.1038/nmeth.3945 · Cited by 395
Mean squared error of prediction as a criterion for evaluating and comparing system models
Wallach, D., & Goffinet, B. (1989) · Ecological Modelling · doi:10.1016/0304-3800(89)90035-5 · Cited by 315
Computed Tomographic Virtual Colonoscopy Computer-Aided Polyp Detection in a Screening Population
Summers, R. M., Yao, J., Pickhardt, P. J., Franaszek, M., Bitter, I., Brickman, D., Krishna, V., & Choi, J. R. (2005) · Gastroenterology · doi:10.1053/j.gastro.2005.08.054 · Cited by 276
Machine learning for outcome prediction of acute ischemic stroke post intra-arterial therapy
Asadi, H., Dowling, R., Yan, B., & Mitchell, P. (2014) · PLoS ONE · doi:10.1371/journal.pone.0088225 · Cited by 218
Self-citationA Cluster-then-label Semi-supervised Learning Approach for Pathology Image Classification
Peikari, M., Salama, S., Nofech-Mozes, S., & Martel, A. L. (2018) · Scientific Reports · doi:10.1038/s41598-018-24876-0 · Cited by 161
Computer-Aided Diagnosis of Lung Cancer and Pulmonary Embolism in Computed Tomography—A Review
Chan, H., Hadjiiski, L. M., Zhou, C., & Sahiner, B. (2008) · Academic Radiology · doi:10.1016/j.acra.2008.01.014 · Cited by 87
Guan, W.-J., Jiang, M., Gao, Y.-H., Li, H., Xu, G., Zheng, J.-P., Chen, R.-C., & Zhong, N.-S. (2016) · The International Journal of Tuberculosis and Lung Disease · doi:10.5588/ijtld.15.0500 · Cited by 47
Moskowitz, C. S. (2017) · JAMA · doi:10.1001/jama.2017.18686 · Cited by 24
Cited by
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443
Chicco, D., Tötsch, N., & Jurman, G. (2021) · BioData Mining · doi:10.1186/s13040-021-00244-z · Cited by 893
Schachner, T., Keller, R., & v Wangenheim, F. (2020) · Journal of Medical Internet Research · doi:10.2196/20701 · Cited by 267
“In addition to the above-mentioned standardized guidelines for research in health care, future research should make use of already existing guidelines for reporting the technical part of AI-based conversational agents used in health care and medicine [72,73].”
BackgroundEllahham, S., Ellahham, N., & Simsekler, M. C. E. (2020) · American Journal of Medical Quality · doi:10.1177/1062860619878515 · Cited by 177
A Scoping Review of Machine Learning in Psychotherapy Research
Aafjes-van Doorn, K., Kamsteeg, C., Bate, J., & Aafjes, M. (2021) · Psychotherapy Research · doi:10.1080/10503307.2020.1808729 · Cited by 179
Metadata
- Authors
- Handelman, G. S., Kok, H. K., Chandra, R. V., Razavi, A. H., Huang, S., Brooks, M., Lee, M. J., & Asadi, H.
- Venue
- American Journal of Roentgenology
- Publisher
- American Roentgen Ray Society
- Published
- 2019-01
- Volume
- 212
- Issue
- 1
- Pages
- 38-43
- ISSN
- 0002-9580 (ISSN-L)
- 0361-803X
- 1546-3141
- OpenAlex
- W2897120700
- Cited by
- 406
- References
- 22
- FWCI
- 4.05
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.