CoChat

How data analysis affects power, reproducibility and biological insight of RNA-seq studies in complex datasets

Open access

The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.

Abstract

The sequencing of the full transcriptome (RNA-seq) has become the preferred choice for the measurement of genome-wide gene expression. Despite its widespread use, challenges remain in RNA-seq data analysis. One often-overlooked aspect is normalization. Despite the fact that a variety of factors or 'batch effects' can contribute unwanted variation to the data, commonly used RNA-seq normalization methods only correct for sequencing depth. The study of gene expression is particularly problematic when it is influenced simultaneously by a variety of biological factors in addition to the one of interest. Using examples from experimental neuroscience, we show that batch effects can dominate the signal of interest; and that the choice of normalization method affects the power and reproducibility of the results. While commonly used global normalization methods are not able to adequately normalize the data, more recently developed RNA-seq normalization can. We focus on one particular method, RUVSeq and show that it is able to increase power and biological insight of the results. Finally, we provide a tutorial outlining the implementation of RUVSeq normalization that is applicable to a broad range of studies as well as meta-analysis of publicly available data.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/nar/gkv736

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. Unwanted variation dominates the signal in RNA-seq studies in experimental neuroscience. PCA plots of gene counts normalized using either upper-quantile (UQ) or FPKM from publicly available datasets from the mouse hippocampus. (A) GSE0261, mRNA-Seq of wild-type (in red) versus knock-out mice (in blue). A severe batch effect is observed in the WT samples (40). (B) GSE0262, small RNA-Seq of wild-type (in red) versus knock-out mice (in blue). A severe batch effect is observed in the WT and KO samples (40). (C) GSE58797, mRNA-seq of mice injected with shRNA to knock down expression of a gene (green), scrambled shRNA (red, controls) and injected with shRNA to knock down expression of a gene and submitted to contextual fear conditioning (FC, blue). A batch effect can be observed in the controls, and there's no separation between FC and naïve injected animals (41). (D) GSE61915, mRNA-seq of young (3 weeks, blue) versus old (24 weeks, red) animals. Proper grouping of treatment samples is observed (42). (E) GSE53380, mRNA-seq of wild-type (control, in red), KO animals (in blue), WT animals following novel-object recognition (NOR, purple) and KO animals following NOR (green). One control sample is an outlier, no separation is observed among all other samples (43). (F) GSE65159, mRNA-seq of animals 2 weeks (2wk,red) and 6 weeks (6wk, blue) following the induction of p25 expression (mouse model of Alzheimer's disease, AD) an their respective controls (green and purple). As expected no difference is observed in time without induction of neurodegeneration, proper separation of samples by treatment is observed in the AD mouse model (44). (G) GSE58343, mRNA-seq of home cage (HC, blue) and fear-conditioned animals (FC, red). Includes pair-end (PE) and single-end (SE) technical replicates, RNA obtained from neuronal dendrites (dend) vs. soma, and RNA following ribosome imuno-precipitation (IP) versus supernatant of the same sample (SN). There is no separation between HC and FC samples, or IP and SN samples (45). (H) GSE44229. mRNA-seq of home-cage (HC, red) versus animals obtained following object location memory (OLM, blue). There's no separation between HC and OLM samples (24).
Figure 1. Unwanted variation dominates the signal in RNA-seq studies in experimental neuroscience. PCA plots of gene counts normalized using either upper-quantile (UQ) or FPKM from publicly available datasets from the mouse hippocampus. (A) GSE0261, mRNA-Seq of wild-type (in red) versus knock-out mice (in blue). A severe batch effect is observed in the WT samples (40). (B) GSE0262, small RNA-Seq of wild-type (in red) versus knock-out mice (in blue). A severe batch effect is observed in the WT and KO samples (40). (C) GSE58797, mRNA-seq of mice injected with shRNA to knock down expression of a gene (green), scrambled shRNA (red, controls) and injected with shRNA to knock down expression of a gene and submitted to contextual fear conditioning (FC, blue). A batch effect can be observed in the controls, and there's no separation between FC and naïve injected animals (41). (D) GSE61915, mRNA-seq of young (3 weeks, blue) versus old (24 weeks, red) animals. Proper grouping of treatment samples is observed (42). (E) GSE53380, mRNA-seq of wild-type (control, in red), KO animals (in blue), WT animals following novel-object recognition (NOR, purple) and KO animals following NOR (green). One control sample is an outlier, no separation is observed among all other samples (43). (F) GSE65159, mRNA-seq of animals 2 weeks (2wk,red) and 6 weeks (6wk, blue) following the induction of p25 expression (mouse model of Alzheimer's disease, AD) an their respective controls (green and purple). As expected no difference is observed in time without induction of neurodegeneration, proper separation of samples by treatment is observed in the AD mouse model (44). (G) GSE58343, mRNA-seq of home cage (HC, blue) and fear-conditioned animals (FC, red). Includes pair-end (PE) and single-end (SE) technical replicates, RNA obtained from neuronal dendrites (dend) vs. soma, and RNA following ribosome imuno-precipitation (IP) versus supernatant of the same sample (SN). There is no separation between HC and FC samples, or IP and SN samples (45). (H) GSE44229. mRNA-seq of home-cage (HC, red) versus animals obtained following object location memory (OLM, blue). There's no separation between HC and OLM samples (24).CC BY · PubMed Central
Figure 2. RUV normalization corrects for unwanted variation in FC data. In red control samples matched for time of day (CC), in blue samples obtained 30 min after memory acquisition (FC), in green samples obtained 30 min after memory retrieval (RT). (A) Relative log expression (RLE) plot of all samples following traditional upper-quartile normalization (UQ). (B) RLE plots following normalization with RUV using negative controls and samples (RUVs). (C) Scatterplot of first two principal components (log-scaled, centered counts) following UQ normalization. The first two PCs explained 66% and 6% of the variance, respectively. (D) Scatterplot of first two principal components following RUVs normalization. The first two PCs explained 19.9% and 13.1% of the variance, respectively. Samples do not cluster according to treatment following UQ normalization but do so after applying RUVs. UQ normalization and RLE and PCA plots were performed using the R/Bioconductor package EDASeq (v. 2.0.0). RUVs normalization was performed using the R/Bioconductor package RUVSeq (v. 1.0.0).
Figure 2. RUV normalization corrects for unwanted variation in FC data. In red control samples matched for time of day (CC), in blue samples obtained 30 min after memory acquisition (FC), in green samples obtained 30 min after memory retrieval (RT). (A) Relative log expression (RLE) plot of all samples following traditional upper-quartile normalization (UQ). (B) RLE plots following normalization with RUV using negative controls and samples (RUVs). (C) Scatterplot of first two principal components (log-scaled, centered counts) following UQ normalization. The first two PCs explained 66% and 6% of the variance, respectively. (D) Scatterplot of first two principal components following RUVs normalization. The first two PCs explained 19.9% and 13.1% of the variance, respectively. Samples do not cluster according to treatment following UQ normalization but do so after applying RUVs. UQ normalization and RLE and PCA plots were performed using the R/Bioconductor package EDASeq (v. 2.0.0). RUVs normalization was performed using the R/Bioconductor package RUVSeq (v. 1.0.0).CC BY · PubMed Central
Figure 3. RUV normalization corrects for unwanted variation in GEO datasets. PCA plots of RUVs normalized gene counts (using all genes as negative controls) from publicly available datasets from the mouse hippocampus. (A) GSE0261, mRNA-Seq of wild-type (in red) versus knock-out mice (in blue). Batch effect no longer evident (40). (B) GSE0262, small RNA-Seq of wild-type (in red) versus knock-out mice (in blue). Batch effect no longer evident (40). (C) GSE58797, mRNA-seq of mice injected with shRNA to knock down expression of a gene (green), scrambled shRNA (red, controls) and injected with shRNA to knock down expression of a gene and submitted to contextual fear conditioning (FC, blue). Batch effect no longer evident (41). (D) GSE61915, mRNA-seq of young (3 weeks, blue) versus old (24 weeks, red) animals. Proper grouping of treatment samples is mantained (42). (E) GSE53380, mRNA-seq of wild-type (control, in red), KO animals (in blue), WT animals following novel-object recognition (NOR, purple) and KO animals following NOR (green). Proper grouping of experimental conditions is improved (43). (F) GSE65159, mRNA-seq of animals 2 weeks (2wk,red) and 6 weeks (6wk, blue) following the induction of p25 expression (mouse model of Alzheimer's disease, AD) an their respective controls (green and purple). As expected no difference is observed in time without induction of neurodegeneration, proper separation of samples by treatment is improved (44). (G) GSE58343, mRNA-seq of home cage (HC, blue) and fear-conditioned animals (FC, red). Includes pair-end (PE) and single-end (SE) technical replicates, RNA obtained from neuronal dendrites (dend) versus soma, and RNA following ribosome imuno-precipitation (IP) versus supernatant of the same sample (SN). Separation separation between HC and FC samples, as well as IP and SN samples is improved (45). (H) GSE44229. mRNA-seq of home-cage (HC, red) versus animals obtained following object location memory (OLM, blue). Batch effect no longer present (24).
Figure 3. RUV normalization corrects for unwanted variation in GEO datasets. PCA plots of RUVs normalized gene counts (using all genes as negative controls) from publicly available datasets from the mouse hippocampus. (A) GSE0261, mRNA-Seq of wild-type (in red) versus knock-out mice (in blue). Batch effect no longer evident (40). (B) GSE0262, small RNA-Seq of wild-type (in red) versus knock-out mice (in blue). Batch effect no longer evident (40). (C) GSE58797, mRNA-seq of mice injected with shRNA to knock down expression of a gene (green), scrambled shRNA (red, controls) and injected with shRNA to knock down expression of a gene and submitted to contextual fear conditioning (FC, blue). Batch effect no longer evident (41). (D) GSE61915, mRNA-seq of young (3 weeks, blue) versus old (24 weeks, red) animals. Proper grouping of treatment samples is mantained (42). (E) GSE53380, mRNA-seq of wild-type (control, in red), KO animals (in blue), WT animals following novel-object recognition (NOR, purple) and KO animals following NOR (green). Proper grouping of experimental conditions is improved (43). (F) GSE65159, mRNA-seq of animals 2 weeks (2wk,red) and 6 weeks (6wk, blue) following the induction of p25 expression (mouse model of Alzheimer's disease, AD) an their respective controls (green and purple). As expected no difference is observed in time without induction of neurodegeneration, proper separation of samples by treatment is improved (44). (G) GSE58343, mRNA-seq of home cage (HC, blue) and fear-conditioned animals (FC, red). Includes pair-end (PE) and single-end (SE) technical replicates, RNA obtained from neuronal dendrites (dend) versus soma, and RNA following ribosome imuno-precipitation (IP) versus supernatant of the same sample (SN). Separation separation between HC and FC samples, as well as IP and SN samples is improved (45). (H) GSE44229. mRNA-seq of home-cage (HC, red) versus animals obtained following object location memory (OLM, blue). Batch effect no longer present (24).CC BY · PubMed Central
Figure 4. Normalization impacts differential expression after contextual fear conditioning. (A) Distribution of unadjusted edgeR p-values for tests of differential expression between FC and CC samples following UQ normalization. (B) Distribution of unadjusted edgeR P-values for tests of differential expression between FC and CC samples following UQ normalization. The distribution of P-values following UQ normalization is far from the expected uniform. RUV returns uniformity to the p-value distribution and increases discovery of differentially expressed genes (genes that have a low P-value). (C) Volcano plot of differential expression (−log10P-value versus log fold change) of UQ normalized samples. (D) Volcano plot of differential expression of RUVs normalized samples. Genes with and FDR <0.01 are highlighted in blue. Positive controls are circled in red, negative controls are circled in green (Table S2). RUV increases the detection of known differentially expressed genes from 60% to 94%. Differential expression analysis was performed using R/Bioconductor package edgeR (v. 3.8.5).
Figure 4. Normalization impacts differential expression after contextual fear conditioning. (A) Distribution of unadjusted edgeR p-values for tests of differential expression between FC and CC samples following UQ normalization. (B) Distribution of unadjusted edgeR P-values for tests of differential expression between FC and CC samples following UQ normalization. The distribution of P-values following UQ normalization is far from the expected uniform. RUV returns uniformity to the p-value distribution and increases discovery of differentially expressed genes (genes that have a low P-value). (C) Volcano plot of differential expression (−log10P-value versus log fold change) of UQ normalized samples. (D) Volcano plot of differential expression of RUVs normalized samples. Genes with and FDR <0.01 are highlighted in blue. Positive controls are circled in red, negative controls are circled in green (Table S2). RUV increases the detection of known differentially expressed genes from 60% to 94%. Differential expression analysis was performed using R/Bioconductor package edgeR (v. 3.8.5).CC BY · PubMed Central
Figure 5. RUV increases concordance of RNA-seq and microarray differential expression following fear conditioning. Y-axis: number of genes in agreement between microarray and RNA-seq data at any given rank. X-axis: differential expression rank (low to high P-value). In red: differentially expressed genes obtained using edgeR for UQ normalized RNA-seq data relative to those detected by microarrays using limma. In blue: differentially expressed genes obtained using edgeR for RUVs normalized RNA-seq data relative to those detected by microarrays using limma. The agreement between technologies on the top 100 differentially expressed genes doubles with RUVs normalization.
Figure 5. RUV increases concordance of RNA-seq and microarray differential expression following fear conditioning. Y-axis: number of genes in agreement between microarray and RNA-seq data at any given rank. X-axis: differential expression rank (low to high P-value). In red: differentially expressed genes obtained using edgeR for UQ normalized RNA-seq data relative to those detected by microarrays using limma. In blue: differentially expressed genes obtained using edgeR for RUVs normalized RNA-seq data relative to those detected by microarrays using limma. The agreement between technologies on the top 100 differentially expressed genes doubles with RUVs normalization.CC BY · PubMed Central
Figure 6. RUV allows removal of laboratory specific effects for combined analysis of gene expression changes following FC and OLM. In red control samples matched for time of day (CC), in blue samples obtained 30 min after memory acquisition (FC), in green samples obtained after object location memory (OLM). (A) Relative log expression (RLE) plot of all samples following upper-quartile normalization (UQ). (B) RLE plots following normalization with RUV using negative controls and samples (RUVs). (C) Scatterplot of first two principal components (log-scaled, centered counts) following UQ normalization. The first two PCs explained 73.4% and 9.6% of the variance, respectively. (D) Scatterplot of first two principal components following RUVs normalization. The first two PCs explained 15.5% and 9.4% of the variance, respectively. Samples cluster according to laboratory following UQ normalization but cluster according to treatment after applying RUVs.
Figure 6. RUV allows removal of laboratory specific effects for combined analysis of gene expression changes following FC and OLM. In red control samples matched for time of day (CC), in blue samples obtained 30 min after memory acquisition (FC), in green samples obtained after object location memory (OLM). (A) Relative log expression (RLE) plot of all samples following upper-quartile normalization (UQ). (B) RLE plots following normalization with RUV using negative controls and samples (RUVs). (C) Scatterplot of first two principal components (log-scaled, centered counts) following UQ normalization. The first two PCs explained 73.4% and 9.6% of the variance, respectively. (D) Scatterplot of first two principal components following RUVs normalization. The first two PCs explained 15.5% and 9.4% of the variance, respectively. Samples cluster according to laboratory following UQ normalization but cluster according to treatment after applying RUVs.CC BY · PubMed Central
Figure 7. Quantitative and qualitative effects of the choice of normalization method in combined analysis of gene expression changes following FC and OLM. (A) Number of genes and enriched KEGG pathways for OLM and FC relative to combined controls following UQ normalization. UQ normalization leads to inferring housekeeping genes as differentially expressed. (B) Number of genes and enriched KEGG pathways for OLM and FC relative to combined controls following RUVs normalization. The apparent regulation of housekeeping genes has been removed.
Figure 7. Quantitative and qualitative effects of the choice of normalization method in combined analysis of gene expression changes following FC and OLM. (A) Number of genes and enriched KEGG pathways for OLM and FC relative to combined controls following UQ normalization. UQ normalization leads to inferring housekeeping genes as differentially expressed. (B) Number of genes and enriched KEGG pathways for OLM and FC relative to combined controls following RUVs normalization. The apparent regulation of housekeeping genes has been removed.CC BY · PubMed Central
Figure 8. Step-by-step outline of the application of RUV to normalization of RNA-seq data.
Figure 8. Step-by-step outline of the application of RUV to normalization of RNA-seq data.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 105

Metadata

Authors
Peixoto, L., Risso, D., Poplawski, S. G., Wimmer, M. E., Speed, T. P., Wood, M. A., & Abel, T.
Affiliations
  • University of California, Berkeley (US)
  • University of Pennsylvania (US)
  • The University of Melbourne (AU)
  • University of California, Irvine (US)
Corresponding author
Abel, Ted
Venue
Nucleic Acids Research
Published
2015-07-21
Volume
43
Issue
16
Pages
7664-7674
ISSN
  • 0305-1048 (ISSN-L)
  • 1362-4962
OpenAlex
W1910607297
Cited by
105
References
46
Citations by year
2026: 6 · 2025: 9 · 2024: 3 · 2023: 15 · 2022: 10 · 2021: 15 · 2020: 9 · 2019: 9 · 2018: 12 · 2017: 5 · 2016: 10 · 2015: 2
FWCI
4.19
Type
journal article
Language
en
Open access
Locations
  • Nucleic Acids Research (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Minerva Access (University of Melbourne) (submittedVersion, other-oa)
  • Padua Research Archive (University of Padova) (submittedVersion, cc-by)
  • eScholarship (California Digital Library) (submittedVersion)
Funders
  • National Institutes of Health
  • Defense Advanced Research Projects Agency (58077LSDRP, 58077, DARPA 58077 LSDRP)
  • National Heart, Lung, and Blood Institute (T32HL007953)
  • National Institute of Mental Health (R01MH101491, R01MH087463)
  • National Institute on Drug Abuse (DA 036984, R01DA036984)
  • National Institute of General Medical Sciences (T32GM008076)
  • National Institute of Neurological Disorders and Stroke (T32NS007413)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
Keywords
RNA-seq normalization, batch effects, statistical power, reproducibility, gene expression, sequencing depth, global normalization, transcriptome sequencing
MeSH terms
Animals, Male, Mice, Inbred C57BL, Neurosciences, Genetic Variation, Reproducibility of Results, Sequence Analysis, RNA, Gene Expression Profiling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.