Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.

Bias in error estimation when using cross-validation for model selection

Open access
Varma, Sudhir, Simon, Richard · 2006 · BMC Bioinformatics

Abstract

Background: Cross-validation (CV) is an effective method for estimating the prediction error of a classifier. Some recent articles have proposed methods for optimizing classifiers by choosing classifier parameter values that minimize the CV error estimate. We have evaluated the validity of using the CV error estimate of the optimized classifier as an estimate of the true error expected on independent data. Results: We used CV to optimize the classification parameters for two kinds of classifiers; Shrunken Centroids and Support Vector Machines (SVM). Random training datasets were created, with no difference in the distribution of the features between the two classes. Using these "null" datasets, we selected classifier parameter values that minimized the CV error estimate. 10-fold CV was used for Shrunken Centroids while Leave-One-Out-CV (LOOCV) was used for the SVM. Independent test data was created to estimate the true error. With "null" and "non null" (with differential expression between the classes) data, we also tested a nested CV procedure, where an inner CV loop is used to perform the tuning of the parameters while an outer CV is used to compute an estimate of the error. The CV error estimate for the classifier with the optimal parameters was found to be a substantially biased estimate of the true error that the classifier would incur on independent data. Even though there is no real difference between the two classes for the "null" datasets, the CV error estimate for the Shrunken Centroid with the optimal parameters was less than 30% on 18.5% of simulated training data-sets. For SVM with optimal parameters the estimated error rate was less than 30% on 38% of "null" data-sets. Performance of the optimized classifiers on the independent test set was no better than chance. The nested CV procedure reduces the bias considerably and gives an estimate of the error that is very close to that obtained on the independent testing set for both Shrunken Centroids and SVM classifiers for "null" and "non-null" data distributions. Conclusion: We show that using CV to compute an error estimate for a classifier that has itself been tuned using CV gives a significantly biased estimate of the true error. Proper use of CV for estimating true error of a classifier developed using a well defined algorithm requires that all steps of the algorithm, including classifier parameter tuning, be repeated in each CV loop. A nested CV procedure provides an almost unbiased estimate of the true error.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/1471-2105-7-91

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Distribution of the CV error estimate and the true error for optimized Shrunken Centroids.
Figure 1 Distribution of the CV error estimate and the true error for optimized Shrunken Centroids.CC BY · PubMed Central
Figure 2 Distribution of the CV error estimate and the true error for optimized Support Vector Machine.
Figure 2 Distribution of the CV error estimate and the true error for optimized Support Vector Machine.CC BY · PubMed Central
Figure 3 Distribution of the nested CV error estimate and true error for optimized Shrunken Centroids nested within a LOOCV loop.
Figure 3 Distribution of the nested CV error estimate and true error for optimized Shrunken Centroids nested within a LOOCV loop.CC BY · PubMed Central
Figure 4 Distribution of the nested CV error estimate and true error for optimized SVM nested within a LOOCV loop.
Figure 4 Distribution of the nested CV error estimate and true error for optimized SVM nested within a LOOCV loop.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 13 references

Cited by

See all 2,485

Metadata

Authors
Varma, S., & Simon, R.
Affiliations
National Cancer Institute (US)
Corresponding author
Varma, Sudhir
Venue
BMC Bioinformatics
Published
2006-02-23
Volume
7
Issue
1
Pages
91-91
ISSN
1471-2105 (ISSN-L)
OpenAlex
W2141007997
Cited by
2,485
References
13
Citations by year
2026: 760 · 2025: 246 · 2024: 146 · 2023: 165 · 2022: 178 · 2021: 179 · 2020: 190 · 2019: 110 · 2018: 96 · 2017: 90 · 2016: 49 · 2015: 47 · 2014: 43 · 2013: 42 · 2012: 35
FWCI
7.08
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • BMC Bioinformatics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
Keywords
prediction error estimation, model selection, nested cross-validation, classifier parameter tuning, support vector machine, leave-one-out cross-validation, 10-fold cross-validation
MeSH terms
Algorithms, Artificial Intelligence, Computer Simulation, Data Interpretation, Statistical, Models, Genetic, Pattern Recognition, Automated, Sensitivity and Specificity, Reproducibility of Results, Models, Statistical, Bias, Oligonucleotide Array Sequence Analysis, Gene Expression Profiling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Bias in error estimation when using cross-validation for model selection. https://app.cochat.ai/papers/10.1186/1471-2105-7-91
The paper
Varma, S., & Simon, R. (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics. https://doi.org/10.1186/1471-2105-7-91

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.