Batch Effect Confounding Leads to Strong Bias in Performance Estimates Obtained by Cross-Validation
Open accessAbstract
BACKGROUND: With the large amount of biological data that is currently publicly available, many investigators combine multiple data sets to increase the sample size and potentially also the power of their analyses. However, technical differences ("batch effects") as well as differences in sample composition between the data sets may significantly affect the ability to draw generalizable conclusions from such studies. FOCUS: The current study focuses on the construction of classifiers, and the use of cross-validation to estimate their performance. In particular, we investigate the impact of batch effects and differences in sample composition between batches on the accuracy of the classification performance estimate obtained via cross-validation. The focus on estimation bias is a main difference compared to previous studies, which have mostly focused on the predictive performance and how it relates to the presence of batch effects. DATA: We work on simulated data sets. To have realistic intensity distributions, we use real gene expression data as the basis for our simulation. Random samples from this expression matrix are selected and assigned to group 1 (e.g., 'control') or group 2 (e.g., 'treated'). We introduce batch effects and select some features to be differentially expressed between the two groups. We consider several scenarios for our study, most importantly different levels of confounding between groups and batch effects. METHODS: We focus on well-known classifiers: logistic regression, Support Vector Machines (SVM), k-nearest neighbors (kNN) and Random Forests (RF). Feature selection is performed with the Wilcoxon test or the lasso. Parameter tuning and feature selection, as well as the estimation of the prediction performance of each classifier, is performed within a nested cross-validation scheme. The estimated classification performance is then compared to what is obtained when applying the classifier to independent data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Batch effects make it difficult to draw conclusions from data across experimental batches.
“These batch effects make it difficult to draw conclusions from data across experimental batches (Leek et al., 2010; Parker and Leek, 2012; Soneson et al., 2014; Nygaard et al., 2016; Caicedo et al., 2017).”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Soneson et al., 2014
Measurements from high-throughput screens are confounded by technical artifacts that arise from differences in the execution of each batch.
“Despite attempts to carefully control experimental variables such as temperature, humidity, and reagent concentration, measurements from these screens are confounded by technical artifacts that arise from differences in the execution of each batch.”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Soneson et al., 2014
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Bias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485 · cited 1× · see citing works
“The performance of each classifier is evaluated on the training data by the means of nested cross-validation [12,18–20] as well as on external data.”
MethodTackling the widespread and critical impact of batch effects in high-throughput data
Leek, J. T., Scharpf, R. B., Bravo, H. C., Simcha, D., Langmead, B., Johnson, W. E., Geman, D., Baggerly, K., & Irizarry, R. A. (2010) · Nature Reviews Genetics · doi:10.1038/nrg2825 · Cited by 2,196 · see citing works
Capturing Heterogeneity in Gene Expression Studies by Surrogate Variable Analysis
Leek, J. T., & Storey, J. D. (2007) · PLoS Genetics · doi:10.1371/journal.pgen.0030161 · Cited by 1,930 · see citing works
MAQC Consortium (2010) · Nature Biotechnology · doi:10.1038/nbt.1665 · Cited by 889 · see citing works
Batch effect removal methods for microarray gene expression data integration: a survey
Lazar, C., Meganck, S., Taminau, J., Steenhoff, D., Coletta, A., Molter, C., Weiss-Solis, D. Y., Duque, R., Bersini, H., & Nowe, A. (2013) · Briefings in Bioinformatics · doi:10.1093/bib/bbs037 · Cited by 333 · see citing works
Flagged first, then by citations
Research works printed but not linked: 27
- [1]Edgar R, Domrachev M, Lash AE (2002) Gene Expression Omnibus: NCBI gene expression and hybridization array data repository. Nucleic Acids Research 30: 207– 10. 11752295 PMC99122
- [2]Brazma A, Parkinson H, Sarkans U, Shojatalab M, Vilo J, et al (2003) ArrayExpress – a public repository for microarray gene expression data at the EBI. Nucleic Acids Research 31: 68– 71. 12519949 PMC165538
- [3]Lukk M, Kapushesky M, Nikkila J, Parkinson H, Goncalves A, et al (2010) A global map of human gene expression. Nat Biotech 28: 322– 324. PMC2974261 20379172
- [4]Brown PO, Botstein D (1999) Exploring the new world of the genome with DNA microarrays. Nature Genetics 21: 33– 37. 9915498
- [5]Scherer A (2009) Batch Effects and Noise in Microarray Experiments: Sources and Solutions. Wiley Series in Probability and Statistics. Wiley. Available: http://books.google.ch/books?id=3-8MsCiPqnkC.
- [8]Luo J, Schumacher M, Scherer A, Sanoudou D, Megherbi D, et al (2010) A comparison of batch effect removal methods for enhancement of prediction performance using MAQC-II microarray gene expression data. Pharmacogenomics J 10: 278– 91. 20676067 PMC2920074
- [9]Kerr MK, Churchill GA (2001) Experimental design for gene expression microarrays. Biostatistics 2: 183– 201. 12933549
- [10]Akey JM, Biswas S, Leek JT, Storey JD (2007) On the design and analysis of gene expression studies in human populations. Nat Genet 39: 807– 808. 17597765
- [11]Parker HS, Leek J (2012) The practical effect of batch on genomic prediction. Statistical Applications in Genetics and Molecular Biology 11: Article– 10. PMC3760371 22611599
- [12]Boulesteix AL, Strobl C (2009) Optimal classifier selection and negative bias in error rate estimation: an empirical study on high-dimensional prediction. BMC Med Res Methodol 9: 85. 20025773 PMC2813849
- [14]Ambroise C, McLachlan GJ (2002) Selection bias in gene extraction on the basis of microarray gene-expression data. Proceedings of the National Academy of Sciences 99: 6562– 6566. PMC124442 11983868
- [15]Radmacher MD, McShane LM, Simon R (2002) A Paradigm for Class Prediction Using Gene Expression Profiles. Journal of Computational Biology 9: 505– 511. 12162889
- [16]Simon R, Radmacher MD, Dobblin K, McShane LM (2003) Pitfalls in the Use of DNA Microarray Data for Diagnostic and Prognostic Classification. Journal of the National Cancer Institute 95: 14– 18. 12509396
- [17]Molinaro AM, Simon R, Pfeiffer RM (2005) Prediction error estimation: a comparison of resampling methods. Bioinformatics 21: 3301– 7. 15905277
- [19]Wood IA, Visscher PM, Mengersen KL (2007) Classification based upon gene expression data: bias and precision of error rates. Bioinformatics 23: 1363– 70. 17392326
- [21]Alter O, Brown PO, Botstein D (2000) Singular value decomposition for genome-wide expression data processing and modeling. Proceedings of the National Academy of Sciences 97: 10101– 10106. PMC27718 10963673
- [22]Benito M, Parker J, Du Q, Wu J, Xiang D, et al (2004) Adjustment of systematic microarray data biases. Bioinformatics 20: 105– 114. 14693816
- [23]Johnson W, Li C, Rabinovic A (2006) Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics 8: 118– 27. 16632515
- [24]Sims AH, Smethurst GJ, Hey Y, Okoniewski MJ, Pepper SD, et al (2008) The removal of multiplicative, systematic bias allows integration of breast cancer gene expression datasets – improving meta-analysis and prediction of prognosis. BMC Med Genomics 1: 42. 18803878 PMC2563019
- [25]Chen C, Grennan K, Badner J, Zhang D, Gershon E, et al (2011) Removing batch effects in analysis of expression microarray data: an evaluation of six batch adjustment methods. PLoS ONE 6: e17238. 21386892 PMC3046121
- [27]Gagnon-Bartsch J, Speed TP (2012) Using control genes to correct for unwanted variation in microarray data. Biostatistics 13: 539– 552. 22101192 PMC3577104
- [28]Van Cutsem E, Labianca R, Bodoky G, Barone C, Aranda E, et al (2009) Randomized Phase III Trial Comparing Biweekly Infusional Fluorouracil/Leucovorin Alone or With Irinotecan in the Adjuvant Treatment of Stage III Colon Cancer: PETACC-3. Journal of Clinical Oncology 27: 3117– 3125. 19451425
- [30]James G, Hastie T, Witten D, Tibshirani R (2013) An Introduction to Statistical Learning: With Applications in R. Springer Texts in Statistics. Springer London, Limited.
- [37]Ben-Hur A, Ong CS, Sonnenburg S, Schölkopf B, Rätsch G (2008) Support vector machines and kernels for computational biology. PLoS Computational Biology 4. PMC2547983 18974822
- [38]Tibshirani R (1996) Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society (Series B) 58: 267– 288.
- [39]Sadanandam A, Lyssiotis CA, Homicsko K, Collisson EA, Gibb WJ, et al (2013) A colorectal cancer classification system that associates cellular phenotype and responses to therapy. Nature Medicine 19: 619– 625. PMC3774607 23584089
- [40]Budinska E, Popovici V, Tejpar S, D'Ario G, Lapique N, et al (2013) Gene expression patterns unveil a new level of molecular heterogeneity in colorectal cancer. The Journal of Pathology 231: 63– 76. 23836465 PMC3840702
Cited by
Sorted by citations
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · see citing works
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477 · see citing works
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518 · see citing works
“These batch effects make it difficult to draw conclusions from data across experimental batches (Leek et al., 2010; Parker & Leek, 2012; Soneson et al., 2014; Nygaard et al., 2016; Caicedo et al., 2017).”
- Authors
- Soneson, C., Gerster, S., & Delorenzi, M.
- Affiliations
- SIB Swiss Institute of Bioinformatics (CH)
- Corresponding author
- Soneson, Charlotte
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-06-26
- Volume
- 9
- Issue
- 6
- Pages
- e100335
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4072626
- OpenAlex
- W2084680372
- Cited by
- 84
- References
- 5
- Citations by year
- 2026: 12 · 2025: 13 · 2024: 10 · 2023: 10 · 2022: 6 · 2021: 7 · 2020: 8 · 2018: 7 · 2017: 5 · 2016: 3 · 2015: 3
- FWCI
- 1.50
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- IRIS (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- SERVAL (Université de Lausanne) (submittedVersion, other-oa)
- Funders
- National Science Foundation
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (135421, 320030)
- Fondation Medic
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Cancer Research › Breast Cancer Treatment Studies
- Keywords
- batch effects, confounding, cross-validation, estimation bias, gene expression data, nested cross-validation, support vector machine, random forest, k-nearest neighbors, logistic regression, LASSO feature selection, Wilcoxon test, data combination
- MeSH terms
- Artificial Intelligence, Statistics as Topic, Reproducibility of Results, Bias, Computational Biology, Gene Expression Profiling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
