The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Bias in error estimation when using cross-validation for model selection
Open accessAbstract
Background: Cross-validation (CV) is an effective method for estimating the prediction error of a classifier. Some recent articles have proposed methods for optimizing classifiers by choosing classifier parameter values that minimize the CV error estimate. We have evaluated the validity of using the CV error estimate of the optimized classifier as an estimate of the true error expected on independent data. Results: We used CV to optimize the classification parameters for two kinds of classifiers; Shrunken Centroids and Support Vector Machines (SVM). Random training datasets were created, with no difference in the distribution of the features between the two classes. Using these "null" datasets, we selected classifier parameter values that minimized the CV error estimate. 10-fold CV was used for Shrunken Centroids while Leave-One-Out-CV (LOOCV) was used for the SVM. Independent test data was created to estimate the true error. With "null" and "non null" (with differential expression between the classes) data, we also tested a nested CV procedure, where an inner CV loop is used to perform the tuning of the parameters while an outer CV is used to compute an estimate of the error. The CV error estimate for the classifier with the optimal parameters was found to be a substantially biased estimate of the true error that the classifier would incur on independent data. Even though there is no real difference between the two classes for the "null" datasets, the CV error estimate for the Shrunken Centroid with the optimal parameters was less than 30% on 18.5% of simulated training data-sets. For SVM with optimal parameters the estimated error rate was less than 30% on 38% of "null" data-sets. Performance of the optimized classifiers on the independent test set was no better than chance. The nested CV procedure reduces the bias considerably and gives an estimate of the error that is very close to that obtained on the independent testing set for both Shrunken Centroids and SVM classifiers for "null" and "non-null" data distributions. Conclusion: We show that using CV to compute an error estimate for a classifier that has itself been tuned using CV gives a significantly biased estimate of the true error. Proper use of CV for estimating true error of a classifier developed using a well defined algorithm requires that all steps of the algorithm, including classifier parameter tuning, be repeated in each CV loop. A nested CV procedure provides an almost unbiased estimate of the true error.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221 · cited 1×
“SVMs were introduced by Vapnik [ 8 ] as linear hyperplanes that separate data belonging to different classes while maximizing the margin, or the distance of the training samples to the linear separating hyperplane.”
Method(2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_5195 · Cited by 19,370
Diagnosis of multiple cancer types by shrunken centroids of gene expression
Tibshirani, R., Hastie, T., Narasimhan, B., & Chu, G. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.082099299 · Cited by 2,930 · cited 2×
“This classifier, originally proposed by Tibshirani et al [5], is an extension of the nearest centroid classifier.”
BackgroundMethodImprovements on Cross-Validation: The 632+ Bootstrap Method
Efron, B., & Tibshirani, R. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474007 · Cited by 1,636 · cited 1×
“632 Bootstrap [10] and Bootstrap CV [11].”
Selection bias in gene extraction on the basis of microarray gene-expression data
Ambroise, C., & McLachlan, G. J. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.102102699 · Cited by 1,480 · cited 1×
“[2], Ambroise and McLachlan [3] and Reunanen [4], this gives a very biased estimate of the true error; not much better than the resubstitution estimate.”
BackgroundPrediction error estimation: a comparison of resampling methods
Molinaro, A. M., Simon, R., & Pfeiffer, R. M. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti499 · Cited by 1,339 · cited 1×
“However, low variance methods can also have a large inherent bias [12] (either positive or negative).”
BackgroundSelf-citationPitfalls in the Use of DNA Microarray Data for Diagnostic and Prognostic Classification
Simon, R., Radmacher, M. D., Dobbin, K., & McShane, L. M. (2003) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/95.1.14 · Cited by 1,006 · cited 1×
“[2], Ambroise and McLachlan [3] and Reunanen [4], this gives a very biased estimate of the true error; not much better than the resubstitution estimate.”
Self-citationIizuka, N., Oka, M., Yamada-Okabe, H., Nishida, M., Maeda, Y., Mori, N., Takao, T., Tamesa, T., Tangoku, A., Tabuchi, H., et al. (2003) · The Lancet · doi:10.1016/s0140-6736(03)12775-4 · Cited by 511
Peng, S., Xu, Q., Ling, X. B., Peng, X., Du, W., & Chen, L. (2003) · FEBS Letters · doi:10.1016/s0014-5793(03)01275-4 · Cited by 218
Cited by
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 1,916
“…should ideally embed any model selection steps into the cross-validation procedure itself, rather than simply picking the model that gives the best cross-validated result (for further explanation, see Cawley & Talbot, 2010; Krstajic, Buturovic, Leahy, & Thomas, 2014; Varma & Simon, 2006).”
BackgroundMethodMachine learning algorithm validation with a limited sample size
Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019) · PLOS ONE · doi:10.1371/journal.pone.0224365 · Cited by 1,901
“Varma and Simon [7] suggest that Nested CV provides almost unbiased performance estimates.”
BackgroundMethodResultInfluentialMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Cross-validation pitfalls when selecting and assessing regression and classification models
Krstajic, D., Buturovic, L. J., Leahy, D. E., & Thomas, S. (2014) · Journal of Cheminformatics · doi:10.1186/1758-2946-6-10 · Cited by 1,090
“It is important to note that both Stone [2] and Varma and Simon [9] use leave-one-out cross-validation, while we use V-fold cross-validation.”
BackgroundMethodResultInfluential
Metadata
- Authors
- Varma, S., & Simon, R.
- Affiliations
- National Cancer Institute (US)
- Corresponding author
- Varma, Sudhir
- Venue
- BMC Bioinformatics
- Published
- 2006-02-23
- Volume
- 7
- Issue
- 1
- Pages
- 91-91
- ISSN
- 1471-2105 (ISSN-L)
- PMC
- PMC1397873
- OpenAlex
- W2141007997
- Cited by
- 2,485
- References
- 13
- Citations by year
- 2026: 760 · 2025: 246 · 2024: 146 · 2023: 165 · 2022: 178 · 2021: 179 · 2020: 190 · 2019: 110 · 2018: 96 · 2017: 90 · 2016: 49 · 2015: 47 · 2014: 43 · 2013: 42 · 2012: 35
- FWCI
- 7.08
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- BMC Bioinformatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- prediction error estimation, model selection, nested cross-validation, classifier parameter tuning, support vector machine, leave-one-out cross-validation, 10-fold cross-validation
- MeSH terms
- Algorithms, Artificial Intelligence, Computer Simulation, Data Interpretation, Statistical, Models, Genetic, Pattern Recognition, Automated, Sensitivity and Specificity, Reproducibility of Results, Models, Statistical, Bias, Oligonucleotide Array Sequence Analysis, Gene Expression Profiling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Bias in error estimation when using cross-validation for model selection. https://app.cochat.ai/papers/10.1186/1471-2105-7-91
- The paper
- Varma, S., & Simon, R. (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics. https://doi.org/10.1186/1471-2105-7-91
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.