Comprehensive evaluation of differential gene expression analysis methods for RNA-seq data
Open accessAbstract
A large number of computational methods have been developed for analyzing differential gene expression in RNA-seq data. We describe a comprehensive evaluation of common methods using the SEQC benchmark dataset and ENCODE data. We consider a number of key features, including normalization, accuracy of differential expression detection and differential expression analysis when one condition has no detectable expression. We find significant differences among the methods, but note that array-based methods adapted to RNA-seq data perform comparably to methods designed for RNA-seq. Our results demonstrate that increasing the number of replicate samples significantly improves detection power over increased sequencing depth.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Single-cell methods incorrectly identified many abundant spike-ins as DE in a dataset with equal concentrations of synthetic mRNAs spiked into each single-cell library.
“We found that single-cell methods incorrectly identified many abundant spike-ins as DE (Fig. 2d-e and Supplementary Fig. 2e-f).”
From Confronting false discoveries in single-cell differential expression (2021) · cites this paper as [21]
The Differential expression analysis section (page unavailable) uses synthetic spike-ins at a 1:1 mixing ratio as true negatives, and the Null model evaluation section reports false-positive DE calls in technical replicates. The paper does not report that single-cell methods called many abundant, equally concentrated spike-ins DE; its reported false positives come from a different comparison. (LLM, conf=0.78)
Experimental — accuracy not yet measuredRecent independent comparison studies have demonstrated that the choice of the method (or even the version of a software package) can markedly affect the outcome of the analysis and that no single method is likely to perform favorably for all datasets.
“Recent independent comparison studies have demonstrated that the choice of the method (or even the version of a software package) can markedly affect the outcome of the analysis and that no single method is likely to perform favorably for all datasets [56, 63, 64]”
From A survey of best practices for RNA-seq data analysis (2016) · cites this paper as [63]
Most comparison studies of differential expression methods have used simulated datasets or spike-in samples with known quantities.
“Most comparison studies have focused on simulated datasets [56, 208, 209] or on samples to which exogenous RNA (‘spike-in’) has been added in known quantities [63, 196].”
From A survey of best practices for RNA-seq data analysis (2016) · cites this paper as [63]
limma performs well under many circumstances and is the fastest to run among the tools.
“Among the tools, limma has been shown to perform well under many circumstances and it is also the fastest to run [56, 63, 64].”
From A survey of best practices for RNA-seq data analysis (2016) · cites this paper as [63]
DESeq and edgeR rank genes similarly but DESeq is often relatively conservative and edgeR too liberal in controlling FDR.
“DESeq and edgeR perform similarly in ranking genes but are often relatively conservative or too liberal, respectively, in controlling FDR [63, 209, 210].”
From A survey of best practices for RNA-seq data analysis (2016) · cites this paper as [63]
Cuffdiff and Cuffdiff2 performed surprisingly poorly in the comparisons.
“Cuffdiff and Cuffdiff2 have performed surprisingly poorly in the comparisons [56, 63].”
From A survey of best practices for RNA-seq data analysis (2016) · cites this paper as [63]
Appropriately transformed and normalized sequencing data can be treated similarly to microarray data for statistical modelling and multiple corrections.
“Appropriately transformed and normalized sequencing data can be treated similarly to microarray data as far as statistical modelling and multiple corrections are concerned175,220.”
From Systems biology and gene networks in neurodevelopmental and neurodegenerative disorders (2015) · cites this paper as [175]
In most experiments, biological variability is greater than technical variability, so biological replicates are of greater value than technical replicates.
“In most experiments, biological variability is greater than technical variability, so biological replicates are of greater value than technical replicates174,175,219.”
From Systems biology and gene networks in neurodevelopmental and neurodegenerative disorders (2015) · cites this paper as [175]
Many guidelines for DGE and network analysis are based on theoretical models and empirical assessments, but most use data lacking the spatial, temporal, or disease-relevant complexities of CNS or post-mortem tissue.
“For DGE and network analysis, there are many studies with guidelines that are based on theoretical models and empirical assessments40,173–176, but most studies use data from experiments that do not have the spatial, temporal or disease-relevant complexities that occur in studies of the CNS or post-mortem tissue.”
From Systems biology and gene networks in neurodevelopmental and neurodegenerative disorders (2015) · cites this paper as [175]
The resulting pipeline gives comparable performance to the best negative binomial-based software packages but with greater speed and reliability for large data sets.
“The resulting pipeline gives comparable performance to the best of the negative binomial-based software packages but with greater speed and reliability for large data sets (10,21).”
From limma powers differential expression analyses for RNA-sequencing and microarray studies (2015) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Robinson, M. D., McCarthy, D. J., & Smyth, G. K. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btp616 · Cited by 41,364 · see citing works
BEDTools: a flexible suite of utilities for comparing genomic features
Quinlan, A. R., & Hall, I. M. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq033 · Cited by 27,437 · see citing works
featureCounts: an efficient general purpose program for assigning sequence reads to genomic features
Liao, Y., Smyth, G. K., & Shi, W. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btt656 · Cited by 27,219 · see citing works
Differential expression analysis for sequence count data
Anders, S., & Huber, W. (2010) · Genome Biology · doi:10.1186/gb-2010-11-10-r106 · Cited by 15,581 · see citing works
Trapnell, C., Williams, B. A., Pertea, G., Mortazavi, A., Kwan, G., van Baren, M. J., Salzberg, S. L., Wold, B. J., & Pachter, L. (2010) · Nature Biotechnology · doi:10.1038/nbt.1621 · Cited by 15,556 · see citing works
Mapping and quantifying mammalian transcriptomes by RNA-Seq
Mortazavi, A., Williams, B. A., McCue, K., Schaeffer, L., & Wold, B. (2008) · Nature Methods · doi:10.1038/nmeth.1226 · Cited by 13,696 · see citing works
RNA-Seq: a revolutionary tool for transcriptomics
Wang, Z., Gerstein, M., & Snyder, M. (2009) · Nature Reviews Genetics · doi:10.1038/nrg2484 · Cited by 13,174 · see citing works
Smyth, G. K. (2004) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1027 · Cited by 11,829 · see citing works
Bolstad, B. M., Irizarry, R. A., Åstrand, M., & Speed, T. P. (2003) · Bioinformatics · doi:10.1093/bioinformatics/19.2.185 · Cited by 8,185 · see citing works
A scaling normalization method for differential expression analysis of RNA-seq data
Robinson, M. D., & Oshlack, A. (2010) · Genome Biology · doi:10.1186/gb-2010-11-3-r25 · Cited by 7,876 · see citing works
voom: precision weights unlock linear model analysis tools for RNA-seq read counts
Law, C. W., Chen, Y., Shi, W., & Smyth, G. K. (2014) · Genome Biology · doi:10.1186/gb-2014-15-2-r29 · Cited by 6,155 · see citing works
McCarthy, D. J., Chen, Y., & Smyth, G. K. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks042 · Cited by 5,195 · see citing works
Landscape of transcription in human cells
Djebali, S., Davis, C. A., Merkel, A., Dobin, A., Lassmann, T., Mortazavi, A., Tanzer, A., Lagarde, J., Lin, W., Schlesinger, F., et al. (2012) · Nature · doi:10.1038/nature11233 · Cited by 5,187 · see citing works
Differential analysis of gene regulation at transcript resolution with RNA-seq
Trapnell, C., Hendrickson, D. G., Sauvageau, M., Goff, L., Rinn, J. L., & Pachter, L. (2013) · Nature Biotechnology · doi:10.1038/nbt.2450 · Cited by 3,492 · see citing works
RNA-seq: An assessment of technical reproducibility and comparison with gene expression arrays
Marioni, J. C., Mason, C. E., Mane, S. M., Stephens, M., & Gilad, Y. (2008) · Genome Research · doi:10.1101/gr.079558.108 · Cited by 2,771 · see citing works
Self-citationThe Transcriptional Landscape of the Yeast Genome Defined by RNA Sequencing
Nagalakshmi, U., Wang, Z., Waern, K., Shou, C., Raha, D., Gerstein, M., & Snyder, M. (2008) · Science · doi:10.1126/science.1158441 · Cited by 2,479 · see citing works
MAQC Consortium, Shi, L., Shi, L., Reid, L. H., Jones, W. D., Shippy, R., Warrington, J. A., Baker, S. C., Collins, P. J., de Longueville, F., et al. (2006) · Nature Biotechnology · doi:10.1038/nbt1239 · Cited by 2,126 · see citing works
Bullard, J. H., Purdom, E., Hansen, K. D., & Dudoit, S. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-94 · Cited by 1,680 · see citing works
Detecting differential usage of exons from RNA-seq data
Anders, S., Reyes, A., & Huber, W. (2012) · Genome Research · doi:10.1101/gr.133744.111 · Cited by 1,545 · see citing works
HTSeq – A Python framework to work with high-throughput sequencing dataPreprint
Anders, S., Pyl, P. T., & Huber, W. K. (2014) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/002824 · Cited by 1,286 · see citing works
Dillies, M., Rau, A., Aubert, J., Hennequet‐Antier, C., Jeanmougin, M., Servant, N., Keime, C., Marot, G., Castel, D., Estellé, J., et al. (2012) · Briefings in Bioinformatics · doi:10.1093/bib/bbs046 · Cited by 1,281 · see citing works
MAQC Consortium (2010) · Nature Biotechnology · doi:10.1038/nbt.1665 · Cited by 889 · see citing works
Moderated statistical tests for assessing differences in tag abundance
Robinson, M. D., & Smyth, G. K. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm453 · Cited by 856 · see citing works
baySeq: Empirical Bayesian methods for identifying differential expression in sequence count data
Hardcastle, T. J., & Kelly, K. A. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-422 · Cited by 755 · see citing works
Evaluation of DNA microarray results with quantitative gene expression platforms
Canales, R. D., Luo, Y., Willey, J. C., Austermiller, B., Bárbácioru, C., Boysen, C., Hunkapiller, K., Jensen, R. V., Knight, C. R., Lee, K. Y., et al. (2006) · Nature Biotechnology · doi:10.1038/nbt1236 · Cited by 615 · see citing works
Transcript length bias in RNA-seq data confounds systems biology
Oshlack, A., & Wakefield, M. J. (2009) · Biology Direct · doi:10.1186/1745-6150-4-14 · Cited by 497 · see citing works
The birth of the Epitranscriptome: deciphering the function of RNA modifications
Saletore, Y., Meyer, K. D., Korlach, J., Vilfan, I. D., Jaffrey, S. R., & Mason, C. E. (2012) · Genome biology · doi:10.1186/gb-2012-13-10-175 · Cited by 485 · see citing works
Self-citationNormalization, testing, and false discovery rate estimation for RNA-sequencing data
Li, J., Witten, D. M., Johnstone, I. M., & Tibshirani, R. (2012) · Biostatistics · doi:10.1093/biostatistics/kxr031 · Cited by 369 · cited 14× · see citing works
“• PoissonSeq (v.1.1.2): No minimum expression mean was applied and the number of permutations was 500.”
BackgroundMethodInfluentialIntegrative analysis of the melanoma transcriptome
Berger, M. F., Levin, J. Z., Vijayendran, K., Sivachenko, A., Adiconis, X., Maguire, J., Johnson, L. A., Robinson, J., Verhaak, R. G., Sougnez, C., et al. (2010) · Genome Research · doi:10.1101/gr.103697.109 · Cited by 305 · see citing works
A comparison of statistical methods for detecting differentially expressed genes from RNA‐seq data
Kvam, V. M., Liu, P., & Si, Y. (2012) · American Journal of Botany · doi:10.3732/ajb.1100340 · Cited by 281 · see citing works
Robles, J. A., Qureshi, S. E., Stephen, S. J., Wilson, S. R., Burden, C. J., & Taylor, J. M. (2012) · BMC Genomics · doi:10.1186/1471-2164-13-484 · Cited by 198 · see citing works
Aban, I., Cutter, G., & Mavinga, N. (2008) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2008.07.034 · Cited by 69 · see citing works
Power Computations for Designing Comparative Poisson Trials
Gail, M. H. (1974) · Biometrics · doi:10.2307/2529645 · Cited by 65 · see citing works
Bioinformatics for High Throughput Sequencing
Rodríguez-Ezpeleta, N., Hackenberg, M., & Aransay, A. (2012) · Springer: New York · doi:10.1007/978-1-4614-0782-9 · Cited by 54 · see citing works
featureCounts: an efficient general-purpose read summarization programPreprint
Liao, Y., Smyth, G. K., & Shi, W. (2013) · arXiv (Cornell University) · Cited by 28 · see citing works
Young, M. D., McCarthy, D. J., Wakefield, M. J., Smyth, G. K., Oshlack, A., & Robinson, M. D. (2011) · Springer eBooks · doi:10.1007/978-1-4614-0782-9_10 · Cited by 10 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
limma powers differential expression analyses for RNA-sequencing and microarray studies
Ritchie, M. E., Phipson, B., Wu, D., Hu, Y., Law, C. W., Shi, W., & Smyth, G. K. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv007 · Cited by 40,458 · see citing works
A survey of best practices for RNA-seq data analysis
Conesa, A., Madrigal, P., Tarazona, S., Gomez-Cabrero, D., Cervera, A., McPherson, A., Szcześniak, M. W., Gaffney, D. J., Elo, L. L., Zhang, X., et al. (2016) · Genome Biology · doi:10.1186/s13059-016-0881-8 · Cited by 2,794 · see citing works
Shen, S., Park, J. W., Lu, Z.-X., Lin, L., Henry, M. D., Wu, Y. N., Zhou, Q., & Xing, Y. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1419161111 · Cited by 2,713 · see citing works
Confronting false discoveries in single-cell differential expression
Squair, J. W., Gautier, M., Kathe, C., Anderson, M. A., James, N. D., Hutson, T. H., Hudelle, R., Qaiser, T., Matson, K. J. E., Barraud, Q., et al. (2021) · doi:10.1101/2021.03.12.435024 · Cited by 1,017 · see citing works
SEQC/MAQC-III Consortium (2014) · Nature Biotechnology · doi:10.1038/nbt.2957 · Cited by 983 · see citing works
Schurch, N. J., Schofield, P., Gierliński, M., Cole, C., Sherstnev, A., Singh, V., Wrobel, N., Gharbi, K., Simpson, G. G., Owen-Hughes, T., et al. (2016) · RNA · doi:10.1261/rna.053959.115 · Cited by 966 · see citing works
Systems biology and gene networks in neurodevelopmental and neurodegenerative disorders
Parikshak, N. N., Gandal, M. J., & Geschwind, D. H. (2015) · Nature Reviews Genetics · doi:10.1038/nrg3934 · Cited by 430 · see citing works
Arthur, J. C., Gharaibeh, R. Z., Mühlbauer, M., Perez-Chanona, E., Uronis, J. M., McCafferty, J., Fodor, A. A., & Jobin, C. (2014) · Nature Communications · doi:10.1038/ncomms5724 · Cited by 392 · see citing works
Robustly detecting differential expression in RNA sequencing data using observation weights
Zhou, X., Lindsay, H., & Robinson, M. D. (2014) · Nucleic Acids Research · doi:10.1093/nar/gku310 · Cited by 387 · see citing works
Sahraeian, S. M. E., Mohiyuddin, M., Sebra, R., Tilgner, H., Afshar, P. T., Au, K. F., Bani Asadi, N., Gerstein, M. B., Wong, W. H., Snyder, M. P., et al. (2017) · Nature Communications · doi:10.1038/s41467-017-00050-4 · Cited by 376 · see citing works
Li, Y., Ge, X., Peng, F., Li, W., & Li, J. J. (2022) · Genome Biology · doi:10.1186/s13059-022-02648-4 · Cited by 303 · see citing works
Power analysis and sample size estimation for RNA-Seq differential expression
Ching, T., Huang, S., & Garmire, L. X. (2014) · RNA · doi:10.1261/rna.046011.114 · Cited by 263 · see citing works
A statistical approach for identifying differential distributions in single-cell RNA-seq experiments
Korthauer, K. D., Chu, L.-F., Newton, M. A., Li, Y., Thomson, J., Stewart, R., & Kendziorski, C. (2016) · Genome Biology · doi:10.1186/s13059-016-1077-y · Cited by 273 · see citing works
A broken promise: microbiome differential abundance methods do not control the false discovery rate
Hawinkel, S., Mattiello, F., Bijnens, L., & Thas, O. (2019) · Briefings in Bioinformatics · doi:10.1093/bib/bbx104 · Cited by 199 · see citing works
Delmans, M., & Hemberg, M. (2015) · doi:10.1101/020735 · Cited by 111 · see citing works
Analytical tools and current challenges in the modern era of neuroepigenomics
Maze, I., Shen, L., Zhang, B., Garcia, B. A., Shao, N., Mitchell, A., Sun, H., Akbarian, S., Allis, C. D., & Nestler, E. J. (2014) · Nature Neuroscience · doi:10.1038/nn.3816 · Cited by 100 · see citing works
ROTS: reproducible RNA-seq biomarker detector—prognostic markers for clear cell renal cell cancer
Seyednasrollah, F., Rantanen, K., Jaakkola, P., & Elo, L. L. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkv806 · Cited by 75 · see citing works
Comparison of methods to detect differentially expressed genes between single-cell populations
Jaakkola, M. K., Seyednasrollah, F., Mehmood, A., & Elo, L. L. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbw057 · Cited by 86 · see citing works
Pepe, G., Braga, D., Renzi, T. A., Villa, A., Bolego, C., D’Avila, F., Barlassina, C., Maggi, A., Locati, M., & Vegeto, E. (2017) · Scientific Reports · doi:10.1038/srep44270 · Cited by 80 · see citing works
subSeq: Determining Appropriate Sequencing Depth Through Efficient Read Subsampling
Robinson, D. G., & Storey, J. D. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu552 · Cited by 57 · see citing works
RNA-Seq Reveals Novel Transcriptional Reorganization in Human Alcoholic Brain
Farris, S. P., & Dayne Mayfield, R. (2014) · International Review of Neurobiology · doi:10.1016/b978-0-12-801105-8.00011-4 · Cited by 49 · see citing works
Swindell, W. R., Xing, X., Voorhees, J. J., Elder, J. T., Johnston, A., & Gudjonsson, J. E. (2014) · Physiological Genomics · doi:10.1152/physiolgenomics.00022.2014 · Cited by 42 · see citing works
Marthandan, S., Menzel, U., Priebe, S., Groth, M., Guthke, R., Platzer, M., Hemmerich, P., Kaether, C., & Diekmann, S. (2016) · Biological Research · doi:10.1186/s40659-016-0095-2 · Cited by 39 · see citing works
- Authors
- Rapaport, F., Khanin, R., Liang, Y., Pirun, M., Krek, A., Zumbo, P., Mason, C. E., Socci, N. D., & Betel, D.
- Affiliations
- Memorial Sloan Kettering Cancer Center (US)
- Weill Cornell Medicine (US)
- Corresponding author
- Betel, Doron
- Venue
- Genome Biology
- Published
- 2013-09-10
- Volume
- 14
- Issue
- 9
- Pages
- R95-R95
- ISSN
- 1465-6906 (ISSN-L)
- 1474-760X
- PMC
- PMC4054597
- OpenAlex
- W2070494252
- Cited by
- 733
- References
- 36
- Citations by year
- 2026: 10 · 2025: 30 · 2024: 46 · 2023: 26 · 2022: 36 · 2021: 61 · 2020: 60 · 2019: 46 · 2018: 69 · 2017: 78 · 2016: 85 · 2015: 109 · 2014: 70 · 2013: 7
- FWCI
- 40.43
- Type
- journal article
- Language
- en
- Update notices
- erratum (2015-11-23)
- Open access
- gold
- doi.org
- Locations
- Genome biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Memorial Sloan-Kettering Cancer Center (CA008748, P30 CA008748)
- Weill Cornell Medical College
- National Cancer Institute (P01CA129243, P50CA140146, P50 CA091629, P30CA008748)
- National Institute of Neurological Disorders and Stroke (R01NS076465)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Keywords
- differential gene expression, RNA-seq, benchmarking, normalization, detection power, sequencing depth, sample replicates, ENCODE
- MeSH terms
- Brain Chemistry, Cell Line, Humans, Nerve Tissue Proteins, RNA, Software, Gene Expression, Sequence Analysis, RNA, Oligonucleotide Array Sequence Analysis, Gene Expression Profiling, High-Throughput Nucleotide Sequencing, Signal-To-Noise Ratio, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
