The DOI matches its Crossref record; 31 of the 32 references it cites were found in a registry or an index.
Machine learning algorithm validation with a limited sample size
Open accessAbstract
Advances in neuroimaging, genomic, motion tracking, eye-tracking and many other technology-based data collection methods have led to a torrent of high dimensional datasets, which commonly have a small number of samples because of the intrinsic high cost of data collection involving human participants. High dimensional data with a small number of samples is of critical importance for identifying biomarkers and conducting feasibility and pilot work, however it can lead to biased machine learning (ML) performance estimates. Our review of studies which have applied ML to predict autistic from non-autistic individuals showed that small sample size is associated with higher reported classification accuracy. Thus, we have investigated whether this bias could be caused by the use of validation methods which do not sufficiently control overfitting. Our simulations show that K-fold Cross-Validation (CV) produces strongly biased performance estimates with small sample sizes, and the bias is still evident with sample size of 1000. Nested CV and train/test split approaches produce robust and unbiased performance estimates regardless of sample size. We also show that feature selection if performed on pooled training and testing data is contributing to bias considerably more than parameter tuning. In addition, the contribution to bias by data dimensionality, hyper-parameter space and number of CV folds was explored, and validation methods were compared with discriminable data. The results suggest how to design robust testing methodologies when working with small datasets and how to interpret the results of other studies based on what validation method was used.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“According to the simulations in Vabalas et al. 99 , the bias introduced by k-fold CV is still substantial even with a sample size of 1000.”
From Machine learning in concrete science: applications, challenges, and best practices (2022) · cites this paper as [99]
The cited source says what the claim attributes to it.
In the citing paper
“Machine learning in medical imaging often faces a fundamental dilemma, namely, the small sample size problem.”
From Federated learning for medical image analysis: A survey (2024) · cites this paper as [9]
In the citing paper
“An alternative to k-fold CV is n × k nested CV, which can produce robust and unbiased performance estimates regardless of the size of datasets 99 .”
From Machine learning in concrete science: applications, challenges, and best practices (2022) · cites this paper as [99]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 1×
“Logistic regression was implemented using Scikit-learn library [18].”
MethodLIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · cited 1×
“SVM was implemented with Libsvm library [17].”
Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Elliott, P., Green, J., Landray, M., et al. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001779 · Cited by 12,728
A training algorithm for optimal margin classifiers
Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130401 · Cited by 11,992 · cited 2×
“One, computationally demanding and complex, where Support Vector Machine (SVM) [12] classifier with Radial Basis Function (RBF) kernel was coupled with Support Vector Machine Recursive Feature Elimination (SVM-RFE) [13] feature selection.”
MethodCross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · cited 1×
“Stone [10] pointed out the importance of separating CV used for model development and CV used for model evaluation.”
MethodLibsvm : A library for support vector machines
Ferrari, V. (2008) · Medical Entomology and Zoology · Cited by 10,122
Gene selection for cancer classification using support vector machines
Guyon, I., Weston, J., Barnhill, S., & Vapnik, V. (2002) · Machine Learning · doi:10.1023/a:1012487302797 · Cited by 9,954 · cited 8×
“SVM-RFE algorithm selects features based on how important they are for an SVM classifier to separate classes.”
InfluentialA review of feature selection techniques in bioinformatics
Saeys, Y., Inza, I., & Larrañaga, P. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm344 · Cited by 5,244
Bias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485 · cited 10×
“Varma and Simon [7] suggest that Nested CV provides almost unbiased performance estimates.”
BackgroundMethodResultInfluentialMachine learning applications in genetics and genomics
Libbrecht, M. W., & Noble, W. S. (2015) · Nature Reviews Genetics · doi:10.1038/nrg3920 · Cited by 2,084 · cited 1×
“Then if number of features is large, which is especially true for neuroimaging and gene expression studies [3, 14, 15], feature selection is performed.”
Background
Cited by
Machine learning in healthcare
Habehh, H., & Gohel, S. (2021) · Current Genomics · doi:10.2174/1389202922666210705124359 · Cited by 544
Federated learning for medical image analysis: A survey
Guan, H., Yap, P.-T., Bozoki, A., & Liu, M. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110424 · Cited by 518
Machine Learning for the Diagnosis of Parkinson's Disease: A Review of Literature
Mei, J., Desrosiers, C., & Frasnelli, J. (2021) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2021.633752 · Cited by 401
“Nevertheless, collecting data from a large pool of participants remains challenging in clinical studies, and data generated are commonly of high dimensionality and small sample size (Vabalas et al., 2019).”
BackgroundEvaluation of a decided sample size in machine learning applications
Rajput, D., Wang, W.-J., & Chen, C.-C. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05156-9 · Cited by 411
“For instance, Vabalas [12] investigated the impact of a range of simulated sub-datasets (20–1000) on the performance of support vector machine (SVM) and logistic regression (LR).”
BackgroundMachine learning in concrete science: applications, challenges, and best practices
Li, Z., Yoon, J., Zhang, R., Rajabipour, F., Srubar III, W. V., Dabo, I., & Radlińska, A. (2022) · npj Computational Materials · doi:10.1038/s41524-022-00810-x · Cited by 388
Metadata
- Authors
- Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J.
- Affiliations
- University of Manchester (GB)
- Corresponding author
- Vabalas, Andrius
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2019-11-07
- Volume
- 14
- Issue
- 11
- Pages
- e0224365
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC6837442
- OpenAlex
- W2981679558
- Cited by
- 1,901
- References
- 32
- Citations by year
- 2026: 497 · 2025: 325 · 2024: 316 · 2023: 280 · 2022: 247 · 2021: 167 · 2020: 66 · 2019: 1
- FWCI
- 55.09
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Research Explorer (The University of Manchester) (submittedVersion, cc-by)
- Funders
- UK Engineering and Physical Sciences Research Council and its Doctoral Training Partnership with the University of Manchester (EP/m507969/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- machine learning validation, small sample size, overfitting, nested cross-validation, train-test split, feature selection, hyperparameter optimization, high-dimensional data, autism classification, biomarker discovery
- MeSH terms
- Machine Learning, Algorithms, Data Interpretation, Statistical, Humans, Sample Size, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Machine learning algorithm validation with a limited sample size. https://app.cochat.ai/papers/10.1371/journal.pone.0224365
- The paper
- Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019). Machine learning algorithm validation with a limited sample size. PLOS ONE. https://doi.org/10.1371/journal.pone.0224365
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.