Assessment of PLSDA Cross Validation
Open accessAbstract
Classifying groups of individuals based on their metabolic profile is one of the main topics in metabolomics research. Due to the low number of individuals compared to the large number of variables, this is not an easy task. PLSDA is one of the data analysis methods used for the classification. Unfortunately this method eagerly overfits the data and rigorous validation is necessary. The validation however is far from straightforward. Is this paper we will discuss a strategy based on cross model validation and permutation testing to validate the classification models. It is also shown that too optimistic results are obtained when the validation is not done properly. Furthermore, we advocate against the use of PLSDA score plots for inference of class differences.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The developed PLS-DA model was validated using leave one out cross validation and 1000 times permutation tests, with quality assessed on R2 and Q2 scores.
“The developed PLS-DA model was validated using the leave one out cross validation method, and its quality was assessed on R2 and Q2 scores [36]. Furthermore, this model was validated using 1000 times permutation tests [36].”
From Comprehensive Metabolomic Fingerprinting Combined with Chemometrics Identifies Species- and Variety-Specific Variation of Medicinal Herbs: An Ocimum Study (2023) · cites this paper as [36]
The Data analysis section (3.4; page unavailable) uses leave-one-out cross-validation only in an inner loop; its main single- and double-cross-validation procedures use six folds. Quality assessment (4.2; page unavailable) assesses Q2, but not R2, and reports 2,000 permutations rather than 1,000. The paper therefore supports the general use of cross-validation, Q2 and permutation testing for PLS-DA, but not the stated combination of methods and counts. (LLM, conf=0.89)
Experimental — accuracy not yet measuredThe developed PLS-DA model was validated using a ten-fold cross-validation method.
“The developed PLS-DA model was validated using a ten-fold cross validation method, and its quality was assessed on R2, Q2, and accuracy scores [26].”
From Rapid metabolic fingerprinting with the aid of chemometric models to identify authenticity of natural medicines: Turmeric, Ocimum, and Withania somnifera study (2023) · cites this paper as [26]
Section 2.3 supports cross-validation as a way to validate PLS-DA models, and Section 2.5 discusses Q² and classification errors as quality measures. But Section 3.4 says this paper used six-fold cross-validation, not ten-fold, and the cited passages do not establish R² as an assessment score. The source therefore supports the general approach, not all the specific methods and scores in the citing sentence. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325 · see citing works
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · see citing works
Journal of the Royal Statistical Society (B)
Fildes, R. A. (1993) · International Journal of Forecasting · doi:10.1016/0169-2070(93)90088-5 · Cited by 3,452 · see citing works
Metabolomics—the link between genotypes and phenotypes
Fiehn, O. (2002) · Plant Molecular Biology · doi:10.1023/a:1013713905833 · Cited by 3,445 · see citing works
Metabolomics — the link between genotypes and phenotypes
Fiehn, O. (2002) · Springer eBooks · doi:10.1007/978-94-010-0448-0_11 · Cited by 3,168 · see citing works
Partial least squares for discrimination
Barker, M. L., & Rayens, W. S. (2003) · Journal of Chemometrics · doi:10.1002/cem.785 · Cited by 2,887 · see citing works
Orthogonal projections to latent structures (O‐PLS)
Trygg, J., & Wold, S. (2002) · Journal of Chemometrics · doi:10.1002/cem.695 · Cited by 2,489 · see citing works
OPLS discriminant analysis: combining the strengths of PLS-DA and SIMCA classification
Bylesjö, M., Rantalainen, M., Cloarec, O., Nicholson, J. K., Holmes, E., & Trygg, J. (2006) · Journal of Chemometrics · doi:10.1002/cem.1006 · Cited by 1,475 · see citing works
Trygg, J., Holmes, E. C., & Lundstedt, T. (2006) · Journal of Proteome Research · doi:10.1021/pr060594q · Cited by 1,276 · see citing works
Arcing classifier (with discussion and a rejoinder by the author)
Breiman, L. (1998) · The Annals of Statistics · doi:10.1214/aos/1024691079 · Cited by 1,104 · see citing works
Cloarec, O., Dumas, M., Craig, A., Barton, R. H., Trygg, J., Hudson, J., Blancher, C., Gauguier, D., Lindon, J. C., Holmes, E. C., et al. (2005) · Analytical Chemistry · doi:10.1021/ac048630x · Cited by 912 · see citing works
Pharmaco-metabonomic phenotyping and personalized drug treatment
Andrew Clayton, T., Lindon, J. C., Cloarec, O., Antti, H., Charuel, C., Hanton, G., Provost, J.-P., Le Net, J.-L., Baker, D., Walley, R. J., et al. (2006) · Nature · doi:10.1038/nature04648 · Cited by 824 · see citing works
Statistical strategies for avoiding false discoveries in metabolomics and related experiments
Broadhurst, D., & Kell, D. B. (2006) · Metabolomics · doi:10.1007/s11306-006-0037-z · Cited by 819 · see citing works
Potential of metabolomics as a functional genomics tool
Bino, R. J., Hall, R. D., Fiehn, O., Kopka, J., Saito, K., Draper, J., Nikolau, B. J., Mendes, P., Roessner, U., Beale, M. H., et al. (2004) · Trends in Plant Science · doi:10.1016/j.tplants.2004.07.004 · Cited by 781 · see citing works
Permutation methods: a distance function approach
(2002) · Choice Reviews Online · doi:10.5860/choice.39-3427 · Cited by 612 · see citing works
Application of genetic algorithm-PLS for feature selection in spectral data sets
Leardi, R. (2000) · Journal of Chemometrics · doi:10.1002/1099-128x(200009/12)14:5/6 · Cited by 564 · see citing works
NMR‐based metabonomic approaches for evaluating physiological influences on biofluid composition
Bollard, M. E., Stanley, E. G., Lindon, J. C., Nicholson, J. K., & Holmes, E. C. (2004) · NMR in Biomedicine · doi:10.1002/nbm.935 · Cited by 473 · see citing works
Mielke, P. W., & Berry, K. J. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3449-2 · Cited by 359 · see citing works
O2‐PLS for qualitative and quantitative analysis in multivariate calibration
Trygg, J. (2002) · Journal of Chemometrics · doi:10.1002/cem.724 · Cited by 242 · see citing works
Reducing over-optimism in variable selection by cross-model validation
Anderssen, E., Dyrstad, K., Westad, F., & Martens, H. A. (2006) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2006.04.021 · Cited by 186 · see citing works
Metabonomics Approach To Determine Metabolic Differences between Green Tea and Black Tea Consumption
van Dorsten, F. A., Daykin, C. A., MULDER, T. P. J., & van Duynhoven, J. (2006) · Journal of Agricultural and Food Chemistry · doi:10.1021/jf061016x · Cited by 170 · see citing works
Assessing the performance of statistical validation tools for megavariate metabolomics data
Rubingh, C. M., Bijlsma, S., Derks, E. P. P. A., Bobeldijk, I., Verheij, E. R., Kochhar, S., & Smilde, A. K. (2006) · Metabolomics · doi:10.1007/s11306-006-0022-6 · Cited by 159 · see citing works
Self-citationBrereton, R. G. (2006) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2006.10.005 · Cited by 152 · see citing works
Microbial metabolomics: replacing trial-and-error by the unbiased selection and ranking of targets
WERF, M. J. V. D., Jellema, R. H., & Hankemeier, T. (2005) · Journal of Industrial Microbiology & Biotechnology · doi:10.1007/s10295-005-0231-4 · Cited by 140 · see citing works
Permutation Tests for Classification
Golland, P., Liang, F., Mukherjee, S., & Panchenko, D. (2005) · Lecture notes in computer science · doi:10.1007/11503415_34 · Cited by 139 · see citing works
Symbiosis of chemometrics and metabolomics: past, present, and future
van der Greef, J., & Smilde, A. K. (2005) · Journal of Chemometrics · doi:10.1002/cem.941 · Cited by 137 · see citing works
Self-citationStatistical validation of classification and calibration models using bootstrapped Latin partitions
Harrington, P. D. B. (2006) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2006.10.010 · Cited by 115 · see citing works
van Ommen, B. (2003) · Nutrition · doi:10.1016/j.nut.2003.09.003 · Cited by 98 · see citing works
Vaidyanathan, S., & Goodacre, R. (2007) · Rapid Communications in Mass Spectrometry · doi:10.1002/rcm.3063 · Cited by 54 · see citing works
Ragazzi, E., Pucciarelli, S., Seraglia, R., et al. (2006) · Journal of Mass Spectrometry · doi:10.1002/jms.1113 · Cited by 25 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Breiman, L. (1998). Arcing classifiers. Annals of Statistics, 26(3), 801-824.
- Martens, H., Dijksterhuis, G. B., & Byrne, D. V. (2000). Power of experimental designs, estimated by Monte Carlo simulation. Journal of Chemometrics, 14(5-6), 441-462.
Cited by
Sorted by citations
Scalbert, A., Brennan, L., Fiehn, O., Hankemeier, T., Kristal, B. S., van Ommen, B., Pujos-Guillot, E., Verheij, E., Wishart, D., & Wopereis, S. (2009) · Metabolomics · doi:10.1007/s11306-009-0168-0 · Cited by 565 · see citing works
Metabolic fate of polyphenols in the human superorganism
van Duynhoven, J., Vaughan, E. E., Jacobs, D. M., A. Kemperman, R., van Velzen, E. J. J., Gross, G., Roger, L. C., Possemiers, S., Smilde, A. K., Doré, J., et al. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1000098107 · Cited by 540 · see citing works
Self-citationbioinformatics tools for mass spectroscopy-based metabolomic data processing and analysis
Sugimoto, M., Kawakami, M., Robert, M., Soga, T., & Tomita, M. (2012) · Current Bioinformatics · doi:10.2174/157489312799304431 · Cited by 307 · see citing works
Promises and pitfalls of untargeted metabolomics
Gertsman, I., & Barshop, B. A. (2018) · Journal of Inherited Metabolic Disease · doi:10.1007/s10545-017-0130-7 · Cited by 271 · see citing works
Multilevel data analysis of a crossover designed human nutritional intervention study
van Velzen, E. J. J., Westerhuis, J. A., van Duynhoven, J. P. M., van Dorsten, F. A., Hoefsloot, H. C. J., Jacobs, D. M., Smit, S., Draijer, R., Kroner, C. I., & Smilde, A. K. (2008) · Journal of Proteome Research · doi:10.1021/pr800145j · Cited by 161 · see citing works
Self-citationDraper, J., Lloyd, A. J., Goodacre, R., & Beckmann, M. (2013) · Metabolomics · doi:10.1007/s11306-012-0449-x · Cited by 144 · see citing works
Metabolic Profiling Identifies Phospholipids as Potential Serum Biomarkers for Schizophrenia
Wang, D., Cheng, S. L., Fei, Q., Gu, H., Raftery, D., Cao, B., Sun, X., Yan, J., Zhang, C., & Wang, J. (2019) · Psychiatry Research · doi:10.1016/j.psychres.2018.12.008 · Cited by 73 · see citing works
Gürdeniz, G., Kristensen, M., Skov, T., & Dragsted, L. O. (2012) · Metabolites · doi:10.3390/metabo2010077 · Cited by 61 · see citing works
Barri, T., Holmer-Jensen, J., Hermansen, K., & Dragsted, L. O. (2012) · Analytica Chimica Acta · doi:10.1016/j.aca.2011.12.065 · Cited by 57 · see citing works
Marzetti, E., Guerra, F., Calvani, R., Marini, F., Biancolillo, A., Gervasoni, J., Primiano, A., Coelho-Júnior, H. J., Landi, F., Bernabei, R., et al. (2020) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2020.564417 · Cited by 45 · see citing works
Backshall, A., Alferez, D., Teichert, F., Wilson, I. D., Wilkinson, R. W., Goodlad, R. A., & Keun, H. C. (2009) · Journal of Proteome Research · doi:10.1021/pr800793w · Cited by 42 · see citing works
Khan, S., Rai, A. K., Singh, A., Singh, S., Dubey, B. K., Lal, R. K., Negi, A. S., Birse, N., Trivedi, P. K., Elliott, C. T., et al. (2023) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2023.04.018 · Cited by 34 · see citing works
Rai, A. K., Khan, S., Kumar, A., Dubey, B. K., Lal, R. K., Tiwari, A., Trivedi, P. K., Elliott, C. T., & Ch, R. (2023) · Metabolites · doi:10.3390/metabo13010122 · Cited by 33 · see citing works
Sen, P., Govaere, O., Sinioja, T., McGlinchey, A., Geng, D., Ratziu, V., Bugianesi, E., Schattenberg, J. M., Vidal-Puig, A., Allison, M., et al. (2022) · iScience · doi:10.1016/j.isci.2022.104949 · Cited by 28 · see citing works
Suberu, J., Gromski, P. S., Nordon, A., & Lapkin, A. (2016) · Journal of Pharmaceutical and Biomedical Analysis · doi:10.1016/j.jpba.2015.10.003 · Cited by 19 · see citing works
Effects of stimulation technique, anatomical region, and time on human sweat lipid mediator profiles
Agrawal, K., Waller, J. D., Pedersen, T. L., & Newman, J. W. (2018) · Prostaglandins & Other Lipid Mediators · doi:10.1016/j.prostaglandins.2017.09.007 · Cited by 10 · see citing works
- Authors
- Westerhuis, J. A., Hoefsloot, H. C. J., Smit, S., Vis, D. J., Smilde, A. K., van Velzen, E. J. J., van Duijnhoven, J. P. M., & van Dorsten, F. A.
- Affiliations
- University of Amsterdam (NL)
- Corresponding author
- Westerhuis, Johan A.
- Venue
- Metabolomics
- Published
- 2008-01-24
- Volume
- 4
- Issue
- 1
- Pages
- 81-89
- ISSN
- 1573-3882 (ISSN-L)
- 1573-3890
- OpenAlex
- W1992483596
- Cited by
- 1,471
- References
- 30
- Citations by year
- 2026: 82 · 2025: 71 · 2024: 90 · 2023: 92 · 2022: 109 · 2021: 116 · 2020: 101 · 2019: 90 · 2018: 96 · 2017: 82 · 2016: 92 · 2015: 70 · 2014: 85 · 2013: 95 · 2012: 81
- FWCI
- 16.16
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Metabolomics (publishedVersion, cc-by-nc)
- UvA-DARE (University of Amsterdam) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Metabolomics and Mass Spectrometry Studies
- Physical Sciences › Chemistry › Analytical Chemistry › Spectroscopy and Chemometric Analyses
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Keywords
- PLS-DA, cross-validation, permutation tests, overfitting, metabolic profile, model validation, score plots, class differences
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
