Classification based upon gene expression data: bias and precision of error rates
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
Motivation: Gene expression data offer a large number of potentially useful predictors for the classification of tissue samples into classes, such as diseased and non-diseased. The predictive error rate of classifiers can be estimated using methods such as cross-validation. We have investigated issues of interpretation and potential bias in the reporting of error rate estimates. The issues considered here are optimization and selection biases, sampling effects, measures of misclassification rate, baseline error rates, two-level external cross-validation and a novel proposal for detection of bias using the permutation mean. Results: Reporting an optimal estimated error rate incurs an optimization bias. Downward bias of 3–5% was found in an existing study of classification based on gene expression data and may be endemic in similar studies. Using a simulated non-informative dataset and two example datasets from existing studies, we show how bias can be detected through the use of label permutations and avoided using two-level external cross-validation. Some studies avoid optimization bias by using single-level cross-validation and a test set, but error rates can be more accurately estimated via two-level cross-validation. In addition to estimating the simple overall error rate, we recommend reporting class error rates plus where possible the conditional risk incorporating prior class probabilities and a misclassification cost matrix. We also describe baseline error rates derived from three trivial classifiers which ignore the predictors. Availability: R code which implements two-level external cross-validation with the PAMR package, experiment code, dataset details and additional figures are freely available for non-commercial use from http://www.maths.qut.edu.au/profiles/wood/permr.jsp Contact: i.wood@qut.edu.au Supplementary information: Supplementary data are available at Bioinformatics online.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Khan, J., Wei, J. S., Ringnér, M., Saal, L. H., Ladanyi, M., Westermann, F., Berthold, F., Schwab, M., Antonescu, C. R., Peterson, C., et al. (2001) · Nature Medicine · doi:10.1038/89044 · Cited by 2,644 · cited 4×
“The results on the Khan and Sharma datasets are detailed in Table 3 and 4. On the Khan dataset, 1-external cv produced mean (standard error) results of 0.00026 (0.00027) for Êrr and 0.00023 (0.00023) for Êa. The mean (standard error) estimates for Êrr and Êa from 2-external cv were 0.00717 (0.0069) and 0.00563 (0.0052…”
BackgroundMethodResultInfluentialBias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485 · cited 2×
“Varma and Simon (2006) investigated the same bias by applying the NSC and SVM to simulated datasets containing two classes of exactly 20 points each.”
BackgroundMethodEstimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300 · cited 4×
“We then give suggestions for improvements and examples of their effectiveness in the large p (number of features or predictors), small n (number of labelled samples) case common in gene expression analyses.”
BackgroundMethodInfluential
Cited by
MAQC Consortium (2010) · Nature Biotechnology · doi:10.1038/nbt.1665 · Cited by 889
Multivariate data analysis for neuroimaging data: overview and application to Alzheimer’s disease
Initiative, T. A. D. N., Habeck, C., & Stern, Y. (2010) · Cell Biochemistry and Biophysics · doi:10.1007/s12013-010-9093-0 · Cited by 127
“As has been pointed out though [41], total prediction error is not a good performance metric in the presence of large imbalances in the class strengths of the training labels.”
Batch Effect Confounding Leads to Strong Bias in Performance Estimates Obtained by Cross-Validation
Soneson, C., Gerster, S., & Delorenzi, M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0100335 · Cited by 84
Calderón, K., Philippot, L., Bizouard, F., Breuil, M.-C., Bru, D., & Spor, A. (2018) · Frontiers in Microbiology · doi:10.3389/fmicb.2018.02721 · Cited by 31
Multi-cohort machine learning identifies predictors of cognitive impairment in Parkinson’s disease
Loo, R. T. J., Pavelka, L., Mangone, G., Khoury, F., Vidailhet, M., Corvol, J.-C., Glaab, E., On behalf of the NCER-PD Consortium, Acharya, G., Aguayo, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01862-1 · Cited by 9
“Model performance was evaluated using a two-level nested CV 71 .”
Metadata
- Authors
- Wood, I. A., Visscher, P. M., & Mengersen, K. L.
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-03-28
- Volume
- 23
- Issue
- 11
- Pages
- 1363-1370
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- OpenAlex
- W2146791205
- Cited by
- 75
- References
- 3
- FWCI
- 3.59
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Classification based upon gene expression data: bias and precision of error rates. https://app.cochat.ai/papers/10.1093/bioinformatics/btm117
- The paper
- Wood, I. A., Visscher, P. M., & Mengersen, K. L. (2007). Classification based upon gene expression data: bias and precision of error rates. Bioinformatics. https://doi.org/10.1093/bioinformatics/btm117
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.