Small sample size effects in statistical pattern recognition: Recommendations for practitioners

Raudys, S.J.; Jain, A.K.

doi:10.1109/34.75512

Cited by 1,384Machine Learning and Data ClassificationImbalanced Data Classification TechniquesStatistical Methods and Applications

Small sample size effects in statistical pattern recognition: Recommendations for practitioners

Raudys, S.J., Jain, A.K.·1991·IEEE Transactions on Pattern Analysis and Machine Intelligence

DOI 10.1109/34.75512

Open this paper in CoChat — free account

Abstract

The effects of sample size on feature selection and error estimation for several types of classifiers are discussed. The focus is on the two-class problem. Classifier design in the context of small design sample size is explored. The estimation of error rates under small test sample size is given. Sample size effects in feature selection are discussed. Recommendations for the choice of learning and test sample sizes are given. In addition to surveying prior work in this area, an emphasis is placed on giving practical advice to designers and users of statistical pattern recognition systems.>

Integrity signals

Registry

Crossref · 10.1109/34.75512

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Medical image analysis faces a fundamental small-sample-size problem.
“Machine learning in medical imaging often faces a fundamental dilemma, namely, the small sample size problem.”
From Federated learning for medical image analysis: A survey (2024) · cites this paper as [8]

Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 21,028 · see citing works
Pattern classification and scene analysis
Grenander, U., Duda, R., & Hart, P. (1974) · A Wiley-Interscience publication · doi:10.2307/1573081 · Cited by 17,399 · see citing works
Pattern classification and scene analysis
Duda, R. O., & Hart, P. E. (1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895 · see citing works
Classification and regression trees
Praagman, J. (1985) · European Journal of Operational Research · doi:10.1016/0377-2217(85)90321-2 · Cited by 10,053 · see citing works
Estimation of Error Rates in Discriminant Analysis
Lachenbruch, P. A., & Mickey, M. R. (1968) · Technometrics · doi:10.1080/00401706.1968.10490530 · Cited by 1,508 · see citing works
Handbook of pattern recognition and image processing
(1986) · Computer Vision Graphics and Image Processing · doi:10.1016/0734-189x(86)90086-1 · Cited by 766 · see citing works
39 Dimensionality and sample size considerations in pattern recognition practice
Jain, A. K., & Chandrasekaran, B. (1982) · Handbook of statistics · doi:10.1016/s0169-7161(82)02042-2 · Cited by 560 · see citing works
Self-citation
The Efficiency of Logistic Regression Compared to Normal Discriminant Analysis
Efron, B. (1975) · Journal of the American Statistical Association · doi:10.1080/01621459.1975.10480319 · Cited by 546 · see citing works
Estimation of Error Rates in Discriminant Analysis
Lachenbruch, P. A., & Mickey, M. R. (1968) · Technometrics · doi:10.2307/1266219 · Cited by 463 · see citing works
Bibliography on estimation of misclassification
Toussaint, G. (1974) · IEEE Transactions on Information Theory · doi:10.1109/tit.1974.1055260 · Cited by 307 · see citing works
Considerations of sample and feature size
Foley, D. H. (1972) · IEEE Transactions on Information Theory · doi:10.1109/tit.1972.1054863 · Cited by 293 · see citing works
Patterns in pattern recognition: 1968-1974
Kanal, L. N. (1974) · IEEE Transactions on Information Theory · doi:10.1109/tit.1974.1055306 · Cited by 282 · see citing works
An Intrinsic Dimensionality Estimator from Near-Neighbor Information
Pettis, K., Bailey, T. A., Jain, A. K., & Dubes, R. C. (1979) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1979.4766873 · Cited by 217 · see citing works
Self-citation
Bootstrap Techniques for Error Estimation
Jain, A. K., Dubes, R. C., & Chen, C.-C. (1987) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1987.4767957 · Cited by 206 · see citing works
Self-citation
On dimensionality and sample size in statistical pattern classification
Kanal, L. N., & Chandrasekaran, B. (1971) · Pattern Recognition · doi:10.1016/0031-3203(71)90013-6 · Cited by 198 · see citing works
Robustness of the linear and quadratic discriminant function to certain types of non‐normality
Lachenbruch, P. A., Sneeringer, C., & Revo, L. T. (1973) · Communications in Statistics · doi:10.1080/03610927308827006 · Cited by 189 · see citing works
Optimization of k nearest neighbor density estimates
Fukunaga, K., & Hostetler, L. D. (1973) · IEEE Transactions on Information Theory · doi:10.1109/tit.1973.1055003 · Cited by 188 · see citing works
Hierarchical Classifier Design Using Mutual Information
Sethi, I. K., & Sarvarayudu, G. P. R. (1982) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1982.4767278 · Cited by 184 · see citing works
Asymptotically optimal discriminant functions for pattern classification
Wolverton, C., & Wagner, T. J. (1969) · IEEE Transactions on Information Theory · doi:10.1109/tit.1969.1054295 · Cited by 174 · see citing works
Posterior Odds for Multivariate Normal Classifications
Geisser, S. (1964) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1964.tb00540.x · Cited by 168 · see citing works
Discrete Discriminant Analysis.
Roncek, D. W., Goldstein, M., & Dillon, W. R. (1979) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2065193 · Cited by 167 · see citing works
On Dimensionality, Sample Size, Classification Error, and Complexity of Classification Algorithm in Pattern Recognition
Raudys, Š., & Pikelis, V. (1980) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1980.4767011 · Cited by 167 · see citing works
Additive estimators for probabilities of correct classification
Glick, N. (1978) · Pattern Recognition · doi:10.1016/0031-3203(78)90029-8 · Cited by 146 · see citing works
35 Use of distance measures, information measures and error bounds in feature evaluation
Ben-Bassat, M. (1982) · Handbook of statistics · doi:10.1016/s0169-7161(82)02038-0 · Cited by 139 · see citing works
Recent advances in error rate estimation
Hand, D. J. (1986) · Pattern Recognition Letters · doi:10.1016/0167-8655(86)90054-1 · Cited by 131 · see citing works
On the optimal number of features in the classification of multivariate Gaussian data
Jain, A. K., & Waller, W. G. (1978) · Pattern Recognition · doi:10.1016/0031-3203(78)90008-0 · Cited by 119 · see citing works
Self-citation
Error estimation in pattern recognition viaL_alpha-distance between posterior density functions
Lissack, T., & Fu, K. (1976) · IEEE Transactions on Information Theory · doi:10.1109/tit.1976.1055512 · Cited by 87 · see citing works
8 Nearest neighbor methods in discrimination
Devroye, L., & Wagner, T. J. (1982) · Handbook of statistics · doi:10.1016/s0169-7161(82)02011-2 · Cited by 82 · see citing works
A note on learning for Gaussian properties
Keehn, D. G. (1965) · IEEE Transactions on Information Theory · doi:10.1109/tit.1965.1053726 · Cited by 68 · see citing works
The bias of the apparent error rate in discriminant analysis
McLachlan, G. J. (1976) · Biometrika · doi:10.1093/biomet/63.2.239 · Cited by 52 · see citing works
Classifier Design with Parzen Windows
Jain, A. K., & Ramaswami, M. D. (1988) · Machine intelligence and pattern recognition · doi:10.1016/b978-0-444-87137-4.50021-7 · Cited by 48 · see citing works
Self-citation
The General Distribution of the Error Rate of a Classification Procedure with Application to Logistic Regression Discrimination
O’Neill, T. J. (1980) · Journal of the American Statistical Association · doi:10.1080/01621459.1980.10477446 · Cited by 47 · see citing works
Small sample size effects in statistical pattern recognition: recommendations for practitioners and open problems
Raudys, Š., & Jain, A. K. (2002) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.1990.118138 · Cited by 43 · see citing works
Self-citation
Assessing the performance of an allocation rule
McLachlan, G. J. (1986) · Computers & Mathematics with Applications · doi:10.1016/0898-1221(86)90079-9 · Cited by 29 · see citing works
3 Large sample approximations and asymptotic expansions of classification statistics
Siotani, M. (1982) · Handbook of statistics · doi:10.1016/s0169-7161(82)02006-9 · Cited by 27 · see citing works
Error Rate Estimation in Discriminant Analysis: Recent Advances
McLachlan, G. J. (1987) · Springer eBooks · doi:10.1007/978-94-017-0653-7_12 · Cited by 26 · see citing works
The efficiency of Efron's “Bootstrap” Approach Applied to Error Rate Estimation in Discriminant Analysis
McLachlan, G. J. (1980) · Journal of Statistical Computation and Simulation · doi:10.1080/00949658008810414 · Cited by 22 · see citing works
6 Nonparametric classification
Broffitt, J. D. (1982) · Handbook of statistics · doi:10.1016/s0169-7161(82)02009-4 · Cited by 21 · see citing works
Determination of optimal dimensionality in statistical pattern classification
Raudys, Š. (1979) · Pattern Recognition · doi:10.1016/0031-3203(79)90036-0 · Cited by 14 · see citing works
The Distributions of the Actual Error Rates in Linear Discriminant Analysis
Sayre, J. W. (1980) · Journal of the American Statistical Association · doi:10.1080/01621459.1980.10477453 · Cited by 13 · see citing works
Unbiased Estimators and Classification Problems for Multivariate Normal Populations
Abusev, R. A., & Lumel’skii, Y. P. (1981) · Theory of Probability and Its Applications · doi:10.1137/1125048 · Cited by 9 · see citing works
21 Logical functions in the problems of empirical prediction
Лбов, Г. С. (1982) · Handbook of statistics · doi:10.1016/s0169-7161(82)02024-0 · Cited by 8 · see citing works
On the accuracy of a bootstrap estimate of the classification error
Raudys, Š. (2003) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.1988.28478 · Cited by 7 · see citing works
Pattern recognition. Comparing techniques by competition
Batchelor, B. G. (1976) · Electronics and Power · doi:10.1049/ep.1976.0389 · Cited by 1 · see citing works

Flagged first, then by citations

Sorted by citations

Learning from Imbalanced Data
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359 · see citing works
The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309 · see citing works
A review of classification algorithms for EEG-based brain–computer interfaces
Lotte, F., Congedo, M., Lécuyer, A., Lamarche, F., & Arnaldi, B. (2007) · Journal of Neural Engineering · doi:10.1088/1741-2560/4/2/r01 · Cited by 2,665 · see citing works
Machine learning algorithm validation with a limited sample size
Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019) · PLOS ONE · doi:10.1371/journal.pone.0224365 · Cited by 1,901 · see citing works
Is your dataset big enough? Sample size requirements when using artificial neural networks for discrete choice analysis
Alwosheel, A., van Cranenburgh, S., & Chorus, C. G. (2018) · Journal of Choice Modelling · doi:10.1016/j.jocm.2018.07.002 · Cited by 524 · see citing works
Federated learning for medical image analysis: A survey
Guan, H., Yap, P.-T., Bozoki, A., & Liu, M. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110424 · Cited by 518 · see citing works
Machine learning in major depression: from classification to treatment outcome prediction
Gao, S., Calhoun, V. D., & Sui, J. (2018) · CNS Neuroscience & Therapeutics · doi:10.1111/cns.13048 · Cited by 370 · see citing works
Sample-size determination methodologies for machine learning in medical imaging research: a systematic review
Balki, I., Amirabadi, A., Levman, J., Martel, A. L., Emersic, Z., Meden, B., Garcia-Pedrero, A., Ramirez, S. C., Kong, D., Moody, A. R., et al. (2019) · Canadian Association of Radiologists Journal · doi:10.1016/j.carj.2019.06.002 · Cited by 299 · see citing works
Extreme Learning Machine-Based Classification of ADHD Using Brain Structural MRI Data
Peng, X., Lin, P., Zhang, T., & Wang, J. (2013) · PLoS ONE · doi:10.1371/journal.pone.0079476 · Cited by 194 · see citing works
A Scoping Review of Machine Learning in Psychotherapy Research
Aafjes-van Doorn, K., Kamsteeg, C., Bate, J., & Aafjes, M. (2021) · Psychotherapy Research · doi:10.1080/10503307.2020.1808729 · Cited by 179 · see citing works
Multivariate pattern classification reveals autonomic and experiential representations of discrete emotions.
Kragel, P. A., & LaBar, K. S. (2013) · Emotion · doi:10.1037/a0031820 · Cited by 134 · see citing works
Overfitting in linear feature extraction for classification of high-dimensional image data
Liu, R., & Gillies, D. F. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2015.11.015 · Cited by 123 · see citing works
Systematic misestimation of machine learning performance in neuroimaging studies of depression
Flint, C., Cearns, M., Opel, N., Redlich, R., Mehler, D. M. A., Emden, D., Winter, N. R., Leenings, R., Eickhoff, S. B., Kircher, T., et al. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-021-01020-7 · Cited by 120 · see citing works
Machine learning in bioinformatics: A brief survey and recommendations for practitioners
Bhaskar, H., Hoyle, D. C., & Singh, S. (2006) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2005.09.002 · Cited by 109 · see citing works
Cortical thickness predicts the first onset of major depression in adolescence
Foland‐Ross, L. C., Sacchet, M. D., Prasad, G., Gilbert, B., Thompson, P. M., & Gotlib, I. H. (2015) · International Journal of Developmental Neuroscience · doi:10.1016/j.ijdevneu.2015.07.007 · Cited by 109 · see citing works
A Comparative Analysis of Early and Late Fusion for the Multimodal Two-Class Problem
Pereira, L. M., Salazar, A., & Vergara, L. (2023) · IEEE Access · doi:10.1109/access.2023.3296098 · Cited by 69 · see citing works
Machine learning classifier using abnormal brain network topological metrics in major depressive disorder
Guo, H., Cao, X., Liu, Z., Li, H., Chen, J., & Zhang, K. (2012) · NeuroReport · doi:10.1097/wnr.0b013e32835a650c · Cited by 46 · see citing works
Quantitative classification of pediatric swallowing through accelerometry
Celeste, M., Azadeh, K., Sejdić, E., Berall, G., & Chau, T. (2012) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-9-34 · Cited by 37 · see citing works
Can shape analysis differentiate free-floating internal carotid artery Thrombus from atherosclerotic plaque in patients evaluated with CTA?for stroke or transient ischemic attack?
Thornhill, R. E., Lum, C., Jaberi, A., Stefanski, P., Torres, C. H., Momoli, F., Petrcich, W., & Dowlatshahi, D. (2014) · Academic Radiology · doi:10.1016/j.acra.2013.11.011 · Cited by 33 · see citing works
Early Identification of Language Disorders Using Natural Language Processing and Machine Learning: Challenges and Emerging Approaches
Lammert, J. M., Roberts, A. C., McRae, K., Batterink, L. J., & Butler, B. E. (2025) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2024_jslhr-24-00515 · Cited by 17 · see citing works
Blood-Based Transcriptomic Biomarkers Are Predictive of Neurodegeneration Rather Than Alzheimer’s Disease
Shvetcov, A., Thomson, S., Spathos, J., Cho, A.-N., Wilkins, H. M., Andrews, S. J., Delerue, F., Couttas, T. A., Issar, J. K., Isik, F., et al. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms241915011 · Cited by 12 · see citing works
SHAP-Based Identification of Potential Acoustic Biomarkers in Patients with Post-Thyroidectomy Voice Disorder
Celepli, S., Bigat, I., Karakas, B., Tezcan, H. M., Yar, M. D., Celepli, P., Aksahin, M. F., Hancerliogullari, O., Yilmaz, Y. F., & Erogul, O. (2025) · Diagnostics · doi:10.3390/diagnostics15162065 · Cited by 6 · see citing works

Authors: Raudys, S. J., & Jain, A. K.
Affiliations: Michigan State University (US)
ORCID: Jain, A.K.
Venue: IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Published: 1991-03
Volume: 13
Issue: 3
Pages: 252-264
ISSN: 0162-8828 (ISSN-L)
DOI: 10.1109/34.75512
OpenAlex: W2002645541
Cited by: 1,384
References: 44
Citations by year: 2026: 30 · 2025: 48 · 2024: 56 · 2023: 63 · 2022: 52 · 2021: 84 · 2020: 69 · 2019: 51 · 2018: 64 · 2017: 45 · 2016: 42 · 2015: 74 · 2014: 49 · 2013: 46 · 2012: 56
FWCI: 7.92
Type: journal article
Language: en
Open access: closed
Licence: ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
Locations: IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
Keywords: statistical pattern recognition, feature selection, error rate estimation, classifier design, binary classification, test sample size, classifier generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Small sample size effects in statistical pattern recognition: Recommendations for practitioners

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks