Classifier design for computer‐aided diagnosis: Effects of finite sample size on the mean performance of classical and neural network classifiers
Open accessThe DOI matches its Crossref record.
Abstract
Classifier design is one of the key steps in the development of computer‐aided diagnosis (CAD) algorithms. A classifier is designed with case samples drawn from the patient population. Generally, the sample size available for classifier design is limited, which introduces variance and bias into the performance of the trained classifier, relative to that obtained with an infinite sample size. For CAD applications, a commonly used performance index for a classifier is the area, under the receiver operating characteristic (ROC) curve. We have conducted a computer simulation study to investigate the dependence of the mean performance, in terms of on design sample size for a linear discriminant and two nonlinear classifiers, the quadratic discriminant and the backpropagation neural network (ANN). The performances of the classifiers were compared for four types of class distributions that have specific properties: multivariate normal distributions with equal covariance matrices and unequal means, unequal covariance matrices and unequal means, and unequal covariance matrices and equal means, and a feature space where the two classes were uniformly distributed in disjoint checkerboard regions. We evaluated the performances of the classifiers in feature spaces of dimensionality ranging from 3 to 15, and design sample sizes from 20 to 800 per class. The dependence of the resubstitution and hold‐out performance on design (training) sample size was investigated. For multivariate normal class distributions with equal covariance matrices, the linear discriminant is the optimal classifier. It was found that its curves can be closely approximated by linear dependences over the range of sample sizes studied. In the feature spaces with unequal covariance matrices where the quadratic discriminant is optimal, the linear discriminant is inferior to the quadratic discriminant or the ANN when the design sample size is large. However, when the design sample is small, a relatively simple classifier, such as the linear discriminant or an ANN with very few hidden nodes, may be preferred because performance bias increases with the complexity of the classifier. In the regime where the classifier performance is dominated by the term, the performance in the limit of infinite sample size can be estimated as the intercept of a linear regression of versus The understanding of the performance of the classifiers under the constraint of a finite design sample size is expected to facilitate the selection of a proper classifier for a given classification task and the design of an efficient resampling scheme.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Deep Learning in Medical Image Analysis
Chan, H.-P., Samala, R. K., Hadjiiski, L. M., & Zhou, C. (2020) · Advances in Experimental Medicine and Biology · doi:10.1007/978-3-030-33128-3_1 · Cited by 747
Self-citationDeep learning in medical imaging and radiation therapy
Sahiner, B., Pezeshk, A., Hadjiiski, L. M., Wang, X., Drukker, K., Cha, K. H., Summers, R. M., & Giger, M. L. (2019) · Medical Physics · doi:10.1002/mp.13264 · Cited by 740
Self-citationAI applications to medical images: From machine learning to deep learning
Castiglioni, I., Rundo, L., Codari, M., Di Leo, G., Salvatore, C., Interlenghi, M., Gallivanone, F., Cozzi, A., D'Amico, N. C., & Sardanelli, F. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.02.006 · Cited by 710
Lachenbruch, P. A. (1967) · Biometrics · doi:10.2307/2528418 · Cited by 499
A review of evidence of health benefit from artificial neural networks in medical intervention
Lisboa, P. J. G. (2002) · Neural Networks · doi:10.1016/s0893-6080(01)00111-3 · Cited by 458
Giger, M. L., Chan, H., & Boone, J. (2008) · Medical Physics · doi:10.1118/1.3013555 · Cited by 311
Self-citationWei, L., Yang, Y., Nishikawa, R. M., & Jiang, Y. (2005) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2004.842457 · Cited by 310
Balki, I., Amirabadi, A., Levman, J., Martel, A. L., Emersic, Z., Meden, B., Garcia-Pedrero, A., Ramirez, S. C., Kong, D., Moody, A. R., et al. (2019) · Canadian Association of Radiologists Journal · doi:10.1016/j.carj.2019.06.002 · Cited by 299
Samala, R. K., Chan, H., Hadjiiski, L., Helvie, M. A., Wei, J., & Cha, K. (2016) · Medical Physics · doi:10.1118/1.4967345 · Cited by 289
Self-citationComputer-Aided Diagnosis Systems for Lung Cancer: Challenges and Methodologies
El-Baz, A., Beache, G. M., Gimel'farb, G., Suzuki, K., Okada, K., Elnakib, A., Soliman, A., & Abdollahi, B. (2013) · International Journal of Biomedical Imaging · doi:10.1155/2013/942353 · Cited by 277
Evaluating Imaging and Computer-aided Detection and Diagnosis Devices at the FDA
Gallas, B. D., Chan, H.-P., D’Orsi, C. J., Dodd, L. E., Giger, M. L., Gur, D., Krupinski, E. A., Metz, C. E., Myers, K. J., Obuchowski, N. A., et al. (2012) · Academic Radiology · doi:10.1016/j.acra.2011.12.016 · Cited by 81
Self-citation
Metadata
- Authors
- Chan, H., Sahiner, B., Wagner, R. F., & Petrick, N.
- Venue
- Medical Physics
- Publisher
- Wiley
- Published
- 1999-12
- Volume
- 26
- Issue
- 12
- Pages
- 2654-2668
- ISSN
- 0094-2405 (ISSN-L)
- 2473-4209
- OpenAlex
- W2015056255
- Cited by
- 146
- References
- 0
- FWCI
- 15.87
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.