CoChat
Sign Up
Cited by 892Gene expression and cancer classificationGenomics and Phylogenetic StudiesSingle-cell and spatial transcriptomics

The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.

A comparison of methods for differential expression analysis of RNA-seq data

Open access
Soneson, Charlotte, Delorenzi, Mauro·2013·BMC Bioinformatics
DOI 10.1186/1471-2105-14-91PMC PMC3608160
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Finding genes that are differentially expressed between conditions is an integral part of understanding the molecular basis of phenotypic variation. In the past decades, DNA microarrays have been used extensively to quantify the abundance of mRNA corresponding to different genes, and more recently high-throughput sequencing of cDNA (RNA-seq) has emerged as a powerful competitor. As the cost of sequencing decreases, it is conceivable that the use of RNA-seq for differential expression analysis will increase rapidly. To exploit the possibilities and address the challenges posed by this relatively new type of data, a number of software packages have been developed especially for differential expression analysis of RNA-seq data. RESULTS: We conducted an extensive comparison of eleven methods for differential expression analysis of RNA-seq data. All methods are freely available within the R framework and take as input a matrix of counts, i.e. the number of reads mapping to each genomic feature of interest in each of a number of samples. We evaluate the methods based on both simulated data and real RNA-seq data. CONCLUSIONS: Very small sample sizes, which are still common in RNA-seq experiments, impose problems for all evaluated methods and any results obtained under such conditions should be interpreted with caution. For larger sample sizes, the methods combining a variance-stabilizing transformation with the 'limma' method for differential expression analysis perform well under many different conditions, as does the nonparametric SAMseq method.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/1471-2105-14-91

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Area under the ROC curve (AUC). Area under the ROC curve (AUC) for the eleven evaluated methods, in simulation studies B0¹²⁵⁰ (panel A), B625⁶²⁵ (panel B), B0⁴⁰⁰⁰ (panel C), B2000²⁰⁰⁰ (panel D), S625⁶²⁵ (panel E) and R625⁶²⁵ (panel F). The boxplots summarize the AUCs obtained across 10 independently simulated instances of each simulation study. Each panel shows the AUCs across three sample sizes (|S1| = |S2| = 2, 5 and 10, respectively, signified by the last number in the tick labels). The methods are ordered according to their median AUC for the largest sample size. When all DE genes were regulated in the same direction, increasing the number of DE genes from 1,250 (panel A) to 4,000 (panel C) impaired the performance of all methods. In contrast, when the DE genes were regulated in different directions (panels B and D), the number of DE genes had much less impact. The variability of the performance of baySeq was much higher when all genes were regulated in the same direction (panels A and C) compared to when the DE genes were regulated in different directions (panels B and D). Including outliers (panels E and F) decreased the AUC for most methods (compare to panel B), but less so for the transformation-based methods (voom+limma and vst+limma) and SAMseq.
Figure 1 Area under the ROC curve (AUC). Area under the ROC curve (AUC) for the eleven evaluated methods, in simulation studies B0¹²⁵⁰ (panel A), B625⁶²⁵ (panel B), B0⁴⁰⁰⁰ (panel C), B2000²⁰⁰⁰ (panel D), S625⁶²⁵ (panel E) and R625⁶²⁵ (panel F). The boxplots summarize the AUCs obtained across 10 independently simulated instances of each simulation study. Each panel shows the AUCs across three sample sizes (|S1| = |S2| = 2, 5 and 10, respectively, signified by the last number in the tick labels). The methods are ordered according to their median AUC for the largest sample size. When all DE genes were regulated in the same direction, increasing the number of DE genes from 1,250 (panel A) to 4,000 (panel C) impaired the performance of all methods. In contrast, when the DE genes were regulated in different directions (panels B and D), the number of DE genes had much less impact. The variability of the performance of baySeq was much higher when all genes were regulated in the same direction (panels A and C) compared to when the DE genes were regulated in different directions (panels B and D). Including outliers (panels E and F) decreased the AUC for most methods (compare to panel B), but less so for the transformation-based methods (voom+limma and vst+limma) and SAMseq.CC BY · PubMed Central
Figure 2 False discovery curves. Representative false discovery curves, depicting the number of false positives encountered among the T top-ranked genes by the eleven evaluated methods, for T between 0 and 1,500. In all cases, there were 5 samples per condition. A: Simulation study B0¹²⁵⁰. B: Simulation study B625⁶²⁵. C: Simulation study B0⁴⁰⁰⁰D: Simulation study B2000²⁰⁰⁰. E: Simulation study S625⁶²⁵F: Simulation study R625⁶²⁵. Some of the curves do not pass through the origin, since many genes obtained the same ranking score and had to be called simultaneously.
Figure 2 False discovery curves. Representative false discovery curves, depicting the number of false positives encountered among the T top-ranked genes by the eleven evaluated methods, for T between 0 and 1,500. In all cases, there were 5 samples per condition. A: Simulation study B0¹²⁵⁰. B: Simulation study B625⁶²⁵. C: Simulation study B0⁴⁰⁰⁰D: Simulation study B2000²⁰⁰⁰. E: Simulation study S625⁶²⁵F: Simulation study R625⁶²⁵. Some of the curves do not pass through the origin, since many genes obtained the same ranking score and had to be called simultaneously.CC BY · PubMed Central
Figure 3 Type I error rates. Type I error rates, for the six methods providing nominal p-values, in simulation studies B0⁰ (panel A), P0⁰ (panel B), S0⁰ (panel C) and R0⁰ (panel D). Letting some counts follow a Poisson distribution (panel B) reduced the type I error rates for TSPM slightly but had overall a small effect. Including outliers with abnormally high counts (panels C and D) had a detrimental effect on the ability to control the type I error for edgeR and NBPSeq, while DESeq became slightly more conservative.
Figure 3 Type I error rates. Type I error rates, for the six methods providing nominal p-values, in simulation studies B0⁰ (panel A), P0⁰ (panel B), S0⁰ (panel C) and R0⁰ (panel D). Letting some counts follow a Poisson distribution (panel B) reduced the type I error rates for TSPM slightly but had overall a small effect. Including outliers with abnormally high counts (panels C and D) had a detrimental effect on the ability to control the type I error for edgeR and NBPSeq, while DESeq became slightly more conservative.CC BY · PubMed Central
Figure 4 True false discovery rates. True false discovery rates (FDR) observed for an imposed FDR threshold of 0.05, for the nine methods returning adjusted p-values or FDR estimates, in simulation studies B0¹²⁵⁰ (panel A), B625⁶²⁵ (panel B), B0⁴⁰⁰⁰ (panel C) B2000²⁰⁰⁰, (panel D), S625⁶²⁵ (panel E) and R625⁶²⁵ (panel F). With only two samples per condition, three of the methods (vst+limma, voom+limma and SAMseq) did not call any DE genes, and the FDR was considered to be undefined.
Figure 4 True false discovery rates. True false discovery rates (FDR) observed for an imposed FDR threshold of 0.05, for the nine methods returning adjusted p-values or FDR estimates, in simulation studies B0¹²⁵⁰ (panel A), B625⁶²⁵ (panel B), B0⁴⁰⁰⁰ (panel C) B2000²⁰⁰⁰, (panel D), S625⁶²⁵ (panel E) and R625⁶²⁵ (panel F). With only two samples per condition, three of the methods (vst+limma, voom+limma and SAMseq) did not call any DE genes, and the FDR was considered to be undefined.CC BY · PubMed Central
Figure 5 Analysis of the Bottomly data set. A: The number of genes found to be significantly DE between the two mouse strains in the Bottomly data set. B-C: Overlap among the set of DE genes found by different methods. D: The average number of genes found to be significantly DE genes when contrasting two subsets of mice from the same strain, in which case we expect no truly DE genes.
Figure 5 Analysis of the Bottomly data set. A: The number of genes found to be significantly DE between the two mouse strains in the Bottomly data set. B-C: Overlap among the set of DE genes found by different methods. D: The average number of genes found to be significantly DE genes when contrasting two subsets of mice from the same strain, in which case we expect no truly DE genes.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 1

  1. Bancroft T, Nettleton D: Estimation of false discovery rate using permutation p-values with different discrete null distributions. Technical Report: Iowa State University; 2009. [ http://www.stat.iastate.edu/preprint/articles/2009-05.pdf ] []

Cited by

Sorted by citations

See all 892

Related

See all

Metadata

Authors
Soneson, C., & Delorenzi, M.
Affiliations
SIB Swiss Institute of Bioinformatics (CH)
Corresponding author
Soneson, Charlotte
ORCID
Venue
BMC Bioinformatics
Publisher
Springer Science and Business Media LLC
Published
2013-03-09
Volume
14
Issue
1
Pages
91-91
ISSN
1471-2105 (ISSN-L)
DOI
10.1186/1471-2105-14-91
PMC
PMC3608160
OpenAlex
W2074414424
Cited by
887
References
34
Citations by year
2026: 24 · 2025: 21 · 2024: 46 · 2023: 45 · 2022: 50 · 2021: 84 · 2020: 66 · 2019: 82 · 2018: 72 · 2017: 83 · 2016: 91 · 2015: 97 · 2014: 101 · 2013: 24
FWCI
46.70
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Bioinformatics (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • IRIS (submittedVersion)
  • SERVAL (Université de Lausanne) (submittedVersion, other-oa)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
Keywords
differential expression analysis, RNA-seq, method comparison, small sample size, variance stabilizing transformation, limma, count data, R package, simulated data
MeSH terms
Animals, Mice, Inbred C57BL, Mice, Inbred DBA, RNA, Messenger, Software, Genome, Sequence Analysis, RNA, DNA, Complementary, Gene Expression Profiling, Genomics, Mice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review