The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
On Splitting Training and Validation Set: A Comparative Study of Cross-Validation, Bootstrap and Systematic Sampling for Estimating the Generalization Performance of Supervised Learning
Open accessAbstract
Model validation is the most important part of building a supervised model. For building a model with good generalization performance one must have a sensible data splitting strategy, and this is crucial for model validation. In this study, we conducted a comparative study on various reported data splitting methods. The MixSim model was employed to generate nine simulated datasets with different probabilities of mis-classification and variable sample sizes. Then partial least squares for discriminant analysis and support vector machines for classification were applied to these datasets. Data splitting methods tested included variants of cross-validation, bootstrapping, bootstrapped Latin partition, Kennard-Stone algorithm (K-S) and sample set partitioning based on joint X – Y distances algorithm (SPXY). These methods were employed to split the data into training and validation sets. The estimated generalization performances from the validation sets were then compared with the ones obtained from the blind test sets which were generated from the same distribution but were unseen by the training/validation procedure used in model construction. The results showed that the size of the data is the deciding factor for the qualities of the generalization performance estimated from the validation set. We found that there was a significant gap between the performance estimated from the validation set and the one from the test set for the all the data splitting methods employed on small datasets. Such disparity decreased when more samples were available for training/validation, and this is because the models were then moving towards approximations of the central limit theory for the simulated datasets used. We also found that having too many or too few samples in the training set had a negative effect on the estimated model performance, suggesting that it is necessary to have a good balance between the sizes of training set and validation set to have a reliable estimation of model performance. We also found that systematic sampling method such as K-S and SPXY generally had very poor estimation of the model performance, most likely due to the fact that they are designed to take the most representative samples first and thus left a rather poorly representative sample set for model performance estimation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
(2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_5195 · Cited by 19,370
Bootstrap methods: another look at the jackknife
Efron, B. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344552 · Cited by 17,602
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Abe, S. (2001) · Springer eBooks · doi:10.1007/978-1-4471-0285-4 · Cited by 3,721
Computer Aided Design of Experiments
Kennard, R. W., & Stone, L. A. (1969) · Technometrics · doi:10.1080/00401706.1969.10490666 · Cited by 3,648
Duda, R. O., Hart, P. E., & Stork, D. G. (2001) · Cited by 2,463
Linear Model Selection by Cross-validation
Shao, J. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476299 · Cited by 1,790
Assessment of PLSDA Cross Validation
Westerhuis, J. A., Hoefsloot, H. C. J., Smit, S., Vis, D. J., Smilde, A. K., van Velzen, E. J. J., van Duijnhoven, J. P. M., & van Dorsten, F. A. (2008) · Metabolomics · doi:10.1007/s11306-007-0099-6 · Cited by 1,471
Chemometrics: Data Analysis for the Laboratory and Chemical Plant
Brereton, R. G. (2003) · Thammasat University Digital Collections · Cited by 1,355
Validation of Regression Models: Methods and Examples
Snee, R. D. (1977) · Technometrics · doi:10.1080/00401706.1977.10489581 · Cited by 1,283
Classification tools in chemistry. Part 1: linear models. PLS-DA
Ballabio, D., & Consonni, V. (2013) · Analytical Methods · doi:10.1039/c3ay40582f · Cited by 1,255
A method for calibration and validation subset partitioning
Galvão, R. K. H., de Araújo, M. C. U., José, G. E., Pontes, M. J. C., Silva, E. C., & Saldanha, T. C. B. (2005) · Talanta · doi:10.1016/j.talanta.2005.03.025 · Cited by 956
Gromski, P. S., Muhamadali, H., Ellis, D. I., Xu, Y., Correa, E., Turner, M. L., & Goodacre, R. (2015) · Analytica Chimica Acta · doi:10.1016/j.aca.2015.02.012 · Cited by 905
Self-citationChemometrics: Data Analysis for the Laboratory and Chemical Plant
Brereton, R. G. (2003) · Wiley eBooks · doi:10.1002/0470863242 · Cited by 842
Statistical strategies for avoiding false discoveries in metabolomics and related experiments
Broadhurst, D. I., & Kell, D. B. (2007) · Metabolomics · doi:10.1007/s11306-006-0037-z · Cited by 817
Poste, G. (2011) · Nature · doi:10.1038/469156a · Cited by 582
A tutorial on support vector machine-based methods for classification problems in chemometrics
Luts, J., Ojeda, F., Van de Plas, R., De Moor, B., Van Huffel, S., & Suykens, J. A. K. (2010) · Analytica Chimica Acta · doi:10.1016/j.aca.2010.03.030 · Cited by 326
Representative subset selection
Daszykowski, M., Walczak, B., & Massart, D. L. (2002) · Analytica Chimica Acta · doi:10.1016/s0003-2670(02)00651-7 · Cited by 304
Molecular phenotyping of a UK population: defining the human serum metabolome
Dunn, W. B., Lin, W., Broadhurst, D., Begley, P., Brown, M., Zelena, E., Vaughan, A. A., Halsall, A., Harding, N., Knowles, J. D., et al. (2015) · Metabolomics · doi:10.1007/s11306-014-0707-1 · Cited by 253
Self-citationSupport Vector Machines: A Recent Method for Classification in Chemometrics
Yun Xu, A., Zomer, S., & Brereton, R. G. (2006) · Critical Reviews in Analytical Chemistry · doi:10.1080/10408340600969486 · Cited by 206
Shao, J. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476934 · Cited by 199
Metabolomics for the masses: The future of metabolomics in a personalized world
K. Trivedi, D., A. Hollywood, K., & Goodacre, R. (2017) · European Journal of Molecular & Clinical Medicine · doi:10.1016/j.nhtm.2017.06.001 · Cited by 192
Self-citation(2001) · Bootstrap Techniques for Signal Processing · doi:10.1017/cbo9780511536717.006 · Cited by 161
MixSim : An R Package for Simulating Data to Study Performance of Clustering Algorithms
Melnykov, V., Chen, W., & Maitra, R. (2012) · Journal of Statistical Software · doi:10.18637/jss.v051.i12 · Cited by 157
Gromski, P. S., Yun Xu, A., Correa, E. S., Ellis, D. I., Turner, M. L., & Goodacre, R. (2014) · Analytica Chimica Acta · doi:10.1016/j.aca.2014.03.039 · Cited by 126
Self-citationInvestigating the influence of data splitting on the predictive ability of QSAR/QSPR models
Puzyn, T., Mostrąg, A., Gajewicz, A., Skrzyński, M., & Worth, A. P. (2011) · Structural Chemistry · doi:10.1007/s11224-011-9757-4 · Cited by 122
Statistical validation of classification and calibration models using bootstrapped Latin partitions
Harrington, P. D. B. (2006) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2006.10.010 · Cited by 120
Multiple Versus Single Set Validation of Multivariate Models to Avoid Mistakes
Harrington, P. D. B. (2017) · Critical Reviews in Analytical Chemistry · doi:10.1080/10408347.2017.1361314 · Cited by 60
Marini, F., Magrì, A. L., Bucci, R., & Magrì, A. (2007) · Analytica Chimica Acta · doi:10.1016/j.aca.2007.08.006 · Cited by 59
Cited by
Alzubaidi, L., Bai, J., Al-Sabaawi, A., Santamaría, J., Albahri, A. S., Al-dabbagh, B. S. N., Fadhel, M. A., Manoufali, M., Zhang, J., Al-Timemy, A. H., et al. (2023) · Journal of Big Data · doi:10.1186/s40537-023-00727-2 · Cited by 907
Khalil, M., McGough, A. S., Pourmirza, Z., Pazhoohesh, M., & Walker, S. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105287 · Cited by 299
Sharma, A., Mukhopadhyay, T., Rangappa, S. M., Siengchin, S., & Kushvaha, V. (2022) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-021-09700-9 · Cited by 285
Machine Learning in Pharmacometrics: Opportunities and Challenges
McComb, M., Bies, R., & Ramanathan, M. (2022) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.14801 · Cited by 128
Metadata
- Authors
- Xu, Y., & Goodacre, R.
- Venue
- Journal of Analysis and Testing
- Published
- 2018-07
- Volume
- 2
- Issue
- 3
- Pages
- 249-262
- ISSN
- 2096-241X (ISSN-L)
- 2509-4696
- OpenAlex
- W2898280479
- Cited by
- 1,030
- References
- 32
- FWCI
- 22.80
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.