RNAontheBENCH: computational and empirical resources for benchmarking RNAseq quantification and differential expression methods
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
RNA sequencing (RNAseq) has become the method of choice for transcriptome analysis, yet no consensus exists as to the most appropriate pipeline for its analysis, with current benchmarks suffering important limitations. Here, we address these challenges through a rich benchmarking resource harnessing (i) two RNAseq datasets including ERCC ExFold spike-ins; (ii) Nanostring measurements of a panel of 150 genes on the same samples; (iii) a set of internal, genetically-determined controls; (iv) a reanalysis of the SEQC dataset; and (v) a focus on relative quantification (i.e. across-samples). We use this resource to compare different approaches to each step of RNAseq analysis, from alignment to differential expression testing. We show that methods providing the best absolute quantification do not necessarily provide good relative quantification across samples, that count-based methods are superior for gene-level relative quantification, and that the new generation of pseudo-alignment-based software performs as well as established methods, at a fraction of the computing time. We also assess the impact of library type and size on quantification and differential expression analysis. Finally, we have created a R package and a web platform to enable the simple and streamlined application of this resource to the benchmarking of future methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2
Love, M. I., Huber, W., & Anders, S. (2014) · Genome Biology · doi:10.1186/s13059-014-0550-8 · Cited by 95,106
Differential analyses for RNA-seq: transcript-level estimates improve gene-level inferences
Soneson, C., Love, M. I., & Robinson, M. D. (2015) · F1000Research · doi:10.12688/f1000research.7563.1 · Cited by 3,929
Direct multiplexed measurement of gene expression with color-coded probe pairs
Geiss, G. K., Bumgarner, R. E., Birditt, B., Dahl, T., Dowidar, N., Dunaway, D. L., Fell, H. P., Ferree, S., George, R. D., Grogan, T., et al. (2008) · Nature Biotechnology · doi:10.1038/nbt1385 · Cited by 2,066 · cited 1×
“92 transcripts in known concentrations, some of which are differentially-expressed across mixes; (ii) a panel of 150 genes representative of the transcriptome (Supplementary Figure S1), including multiple isoforms for some genes, measured in the same samples by the highly reproducible Nanostring nCounter technology (2…”
MethodAncient DNA and the rewriting of human history: be sparing with Occam’s razor
Haber, M., Mezzavilla, M., Xue, Y., & Tyler-Smith, C. (2016) · Genome Biology · doi:10.1186/s13059-015-0866-z · Cited by 1,924
MapSplice: Accurate mapping of RNA-seq reads for splice junction discovery
Wang, K., Singh, D., Zeng, Z., Coleman, S. J., Huang, Y., Savich, G. L., He, X., Mieczkowski, P., Grimm, S. A., Perou, C. M., et al. (2010) · Nucleic Acids Research · doi:10.1093/nar/gkq622 · Cited by 1,193
SEQC/MAQC-III Consortium (2014) · Nature Biotechnology · doi:10.1038/nbt.2957 · Cited by 983
A comparison of methods for differential expression analysis of RNA-seq data
Soneson, C., & Delorenzi, M. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-91 · Cited by 887 · cited 1×
“Regarding DEA, in addition to a comparison of statistical models (3), previous contributions have for instance studied the effect of ‘zero-count genes’ on some methods (6,15), and shown that isoform filtering can improve differential transcript usage (16).”
MethodComprehensive evaluation of differential gene expression analysis methods for RNA-seq data
Rapaport, F., Khanin, R., Liang, Y., Pirun, M., Krek, A., Zumbo, P., Mason, C. E., Socci, N. D., & Betel, D. (2013) · Genome Biology · doi:10.1186/gb-2013-14-9-r95 · Cited by 733 · cited 1×
“Regarding DEA, in addition to a comparison of statistical models (3), previous contributions have for instance studied the effect of ‘zero-count genes’ on some methods (6,15), and shown that isoform filtering can improve differential transcript usage (16).”
MethodPatro, R., Mount, S. M., & Kingsford, C. (2014) · Nature Biotechnology · doi:10.1038/nbt.2862 · Cited by 586
Systematic evaluation of spliced alignment programs for RNA-seq data
The RGASP Consortium, Engström, P. G., Steijger, T., Sipos, B., Grant, G. R., Kahles, A., Rätsch, G., Goldman, N., Hubbard, T. J., Harrow, J., et al. (2013) · Nature Methods · doi:10.1038/nmeth.2722 · Cited by 513 · cited 1×
“Several benchmarking efforts (1–12) have allowed their comparison, showing a fairly high reproducibility of the most popular methods (9), while highlighting their respective strengths and weaknesses (see especially (1) for alignment and (3) for differential expression).”
Background
Cited by
Differential analysis of RNA-seq incorporating quantification uncertainty
Pimentel, H., Bray, N. L., Puente, S., Melsted, P., & Pachter, L. (2017) · Nature Methods · doi:10.1038/nmeth.4324 · Cited by 1,534
RNA-Seq differential expression analysis: An extended review and a software tool
Costa-Silva, J., Domingues, D., & Lopes, F. M. (2017) · PLOS ONE · doi:10.1371/journal.pone.0190152 · Cited by 626
Gabriele, M., Vulto-van Silfhout, A. T., Germain, P.-L., Vitriolo, A., Kumar, R., Douglas, E., Haan, E., Kosaki, K., Takenouchi, T., Rauch, A., et al. (2017) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2017.05.006 · Cited by 163
Zanella, M., Vitriolo, A., Andirko, A., Martins, P. T., Sturm, S., O’Rourke, T., Laugsch, M., Malerba, N., Skaros, A., Trattaro, S., et al. (2019) · Science Advances · doi:10.1126/sciadv.aaw7908 · Cited by 101
“91 to calculate read counts and transcripts per million in a transcript- and gene-wise fashion, using the quasi-mapping offline algorithm (54) on the GRCh38 (National Center for Biotechnology Information) database.”
MethodSelf-citationGermain, P.-L., & Testa, G. (2017) · Stem Cell Reports · doi:10.1016/j.stemcr.2017.05.012 · Cited by 94
“With the exception of the mixed-models approaches (Figures 4A, 4B, and 6), all differential expression analysis was performed with edgeR v.3.12.1, which has among the most robust performances (Germain et al., 2016).”
MethodInfluentialSelf-citation
Metadata
- Authors
- Germain, P.-L., Vitriolo, A., Adamo, A., Laise, P., Das, V., & Testa, G.
- Venue
- Nucleic Acids Research
- Publisher
- Oxford University Press (OUP)
- Published
- 2016-05-17
- Volume
- 44
- Issue
- 11
- Pages
- 5054-5067
- ISSN
- 0305-1048 (ISSN-L)
- 1362-4962
- OpenAlex
- W2399463958
- Cited by
- 44
- References
- 14
- FWCI
- 3.31
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: RNAontheBENCH: computational and empirical resources for benchmarking RNAseq quantification and differential expression methods. https://app.cochat.ai/papers/10.1093/nar/gkw448
- The paper
- Germain, P.-L., Vitriolo, A., Adamo, A., Laise, P., Das, V., & Testa, G. (2016). RNAontheBENCH: computational and empirical resources for benchmarking RNAseq quantification and differential expression methods. Nucleic Acids Research. https://doi.org/10.1093/nar/gkw448
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.