Evaluation of statistical methods for normalization and differential expression in mRNA-Seq experiments
Open accessAbstract
BACKGROUND: High-throughput sequencing technologies, such as the Illumina Genome Analyzer, are powerful new tools for investigating a wide range of biological and medical questions. Statistical and computational methods are key for drawing meaningful and accurate conclusions from the massive and complex datasets generated by the sequencers. We provide a detailed evaluation of statistical methods for normalization and differential expression (DE) analysis of Illumina transcriptome sequencing (mRNA-Seq) data. RESULTS: We compare statistical methods for detecting genes that are significantly DE between two types of biological samples and find that there are substantial differences in how the test statistics handle low-count genes. We evaluate how DE results are affected by features of the sequencing platform, such as, varying gene lengths, base-calling calibration method (with and without phi X control lane), and flow-cell/library preparation effects. We investigate the impact of the read count normalization method on DE results and show that the standard approach of scaling by total lane counts (e.g., RPKM) can bias estimates of DE. We propose more general quantile-based normalization procedures and demonstrate an improvement in DE detection. CONCLUSIONS: Our results have significant practical and methodological implications for the design and analysis of mRNA-Seq experiments. They highlight the importance of appropriate statistical methods for normalization and DE inference, to account for features of the sequencing platform that could impact the accuracy of results. They also reveal the need for further research in the development of statistical and computational methods for mRNA-Seq.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Normalization methods based on total or effective counts tend to perform poorly when samples have heterogeneous transcript distributions, that is, when highly and differentially expressed features can skew the count distribution.
“These approaches rely on normalizing methods that are based on total or effective counts, and tend to perform poorly when samples have heterogeneous transcript distributions, that is, when highly and differentially expressed features can skew the count distribution [45, 46].”
From A survey of best practices for RNA-seq data analysis (2016) · cites this paper as [45]
While reproducibility in RNA-seq is usually high at the sequencing level, other steps such as RNA extraction and library preparation are noisier and may introduce biases that can be minimized by good experimental procedures.
“While reproducibility in RNA-seq is usually high at the level of sequencing [1, 45], other steps such as RNA extraction and library preparation are noisier and may introduce biases in the data that can be minimized by adopting good experimental procedures (Box 2).”
From A survey of best practices for RNA-seq data analysis (2016) · cites this paper as [45]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Ultrafast and memory-efficient alignment of short DNA sequences to the human genome.
Langmead, B., Trapnell, C., Pop, M., & Salzberg, S. L. (2009) · Genome Biology · doi:10.1186/gb-2009-10-3-r25 · Cited by 21,849 · cited 1× · see citing works
“We used Bowtie to map reads to the genome (GRCh37 assembly) [ 16 ].”
Method(2005) · Springer eBooks · doi:10.1007/0-387-27255-0_9 · Cited by 20,333 · see citing works
Mapping and quantifying mammalian transcriptomes by RNA-Seq
Mortazavi, A., Williams, B. A., McCue, K., Schaeffer, L., & Wold, B. (2008) · Nature Methods · doi:10.1038/nmeth.1226 · Cited by 13,696 · cited 6× · see citing works
“These high-throughput sequencing technologies have already been applied to monitor genome-wide transcription levels (mRNA-Seq), DNA-protein interactions (ChIP-Seq), chromatin structure, and DNA methylation [1-9].”
BackgroundMethodInfluentialSmyth, G. K. (2004) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1027 · Cited by 11,829 · see citing works
Exploration, normalization, and summaries of high density oligonucleotide array probe level data
Irizarry, R. A. (2003) · Biostatistics · doi:10.1093/biostatistics/4.2.249 · Cited by 10,603 · see citing works
Base-Calling of Automated Sequencer Traces Using Phred. I. Accuracy AssessmentPreprint
Ewing, B., Hillier, L., Wendl, M. C., & Green, P. (1998) · Genome Research · doi:10.1101/gr.8.3.175 · Cited by 6,993 · see citing works
Base-Calling of Automated Sequencer Traces Using <i>Phred.</i> II. Error Probabilities
Ewing, B., & Green, P. (1998) · Genome Research · doi:10.1101/gr.8.3.186 · Cited by 5,439 · cited 1× · see citing works
“quality scores for the other seven lanes [[15], Supplementary Information, p. 7]. Bustard can also be run using the auto-calibration method, which scores basecalls in a manner similar to the phred base-caller [ 17 ] and does not require a control lane per flow-cell.”
MethodAlternative isoform regulation in human tissue transcriptomes
Wang, E. T., Sandberg, R., Luo, S., Khrebtukova, I., Zhang, L., Mayr, C., Kingsmore, S. F., Schroth, G. P., & Burge, C. B. (2008) · Nature · doi:10.1038/nature07509 · Cited by 5,181 · see citing works
Accurate whole human genome sequencing using reversible terminator chemistry
Bentley, D. R., Balasubramanian, S., Swerdlow, H. P., Smith, G. P., Milton, J., Brown, C. G., Hall, K. P., Evers, D. J., Barnes, C. L., Bignell, H. R., et al. (2008) · Nature · doi:10.1038/nature07517 · Cited by 3,684 · cited 1× · see citing works
“quality scores for the other seven lanes [[15], Supplementary Information, p.”
BackgroundRNA-seq: An assessment of technical reproducibility and comparison with gene expression arrays
Marioni, J. C., Mason, C. E., Mane, S. M., Stephens, M., & Gilad, Y. (2008) · Genome Research · doi:10.1101/gr.079558.108 · Cited by 2,771 · cited 7× · see citing works
“These high-throughput sequencing technologies have already been applied to monitor genome-wide transcription levels (mRNA-Seq), DNA-protein interactions (ChIP-Seq), chromatin structure, and DNA methylation [1-9].”
BackgroundMethodResultInfluentialThe Transcriptional Landscape of the Yeast Genome Defined by RNA Sequencing
Nagalakshmi, U., Wang, Z., Waern, K., Shou, C., Raha, D., Gerstein, M., & Snyder, M. (2008) · Science · doi:10.1126/science.1158441 · Cited by 2,479 · cited 1× · see citing works
“These high-throughput sequencing technologies have already been applied to monitor genome-wide transcription levels (mRNA-Seq), DNA-protein interactions (ChIP-Seq), chromatin structure, and DNA methylation [1-9].”
MethodMAQC Consortium, Shi, L., Shi, L., Reid, L. H., Jones, W. D., Shippy, R., Warrington, J. A., Baker, S. C., Collins, P. J., de Longueville, F., et al. (2006) · Nature Biotechnology · doi:10.1038/nbt1239 · Cited by 2,126 · see citing works
Substantial biases in ultra-short read data sets from high-throughput DNA sequencing
Dohm, J. C., Lottaz, C., Borodina, T., & Himmelbauer, H. (2008) · Nucleic Acids Research · doi:10.1093/nar/gkn425 · Cited by 1,088 · cited 2× · see citing works
“0 arrays (GSE5350, samples AFX_1_ [A-B] [1-5]; see [Additional file 2: Supplemental Sections S1.”
Moderated statistical tests for assessing differences in tag abundance
Robinson, M. D., & Smyth, G. K. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm453 · Cited by 856 · cited 1× · see citing works
“The use of negative binomial or empirical Bayes methods, as described in the SAGE literature [21,22], may be sensible in such settings of increased variability.”
Method't Hoen, P. A. C., Ariyurek, Y., Thygesen, H. H., Vreugdenhil, E., Vossen, R. H. A. M., de Menezes, R. X., Boer, J. M., van Ommen, G.-J. B., & den Dunnen, J. T. (2008) · Nucleic Acids Research · doi:10.1093/nar/gkn705 · Cited by 742 · cited 2× · see citing works
“literature include t-statistics with square root-transformed standard errors and Bayesian statistics based on the Beta-Binomial distribution (Hoen et al., 2008).”
MethodEvaluation of DNA microarray results with quantitative gene expression platforms
Canales, R. D., Luo, Y., Willey, J. C., Austermiller, B., Bárbácioru, C., Boysen, C., Hunkapiller, K., Jensen, R. V., Knight, C. R., Lee, K. Y., et al. (2006) · Nature Biotechnology · doi:10.1038/nbt1236 · Cited by 615 · cited 5× · see citing works
“Here, we evaluate a variety of normalization procedures and focus on two main questions: (1) Does the normalization improve DE detection (sensitivity)? (2) Does the normalization result in low technical variability across replicates (specificity)? To assess DE detection, we rely on the qRT-PCR data of [13] as a gold-s…”
BackgroundMethodInfluentialHigh-resolution mapping of copy-number alterations with massively parallel sequencing
Chiang, D. Y., Getz, G. A., Jaffe, D. B., O'Kelly, M., Zhao, X., Carter, S. L., Russ, C., Nusbaum, C., Meyerson, M. M., & Lander, E. S. (2008) · Nature Methods · doi:10.1038/nmeth.1276 · Cited by 510 · cited 2× · see citing works
“These high-throughput sequencing technologies have already been applied to monitor genome-wide transcription levels (mRNA-Seq), DNA-protein interactions (ChIP-Seq), chromatin structure, and DNA methylation [ 1-9 ].”
MethodInfluentialTranscript length bias in RNA-seq data confounds systems biology
Oshlack, A., & Wakefield, M. J. (2009) · Biology Direct · doi:10.1186/1745-6150-4-14 · Cited by 497 · cited 3× · see citing works
“Under the assumptions presented in [11], with the unweighted t-statistics and using the same cutoff across genes, power increases with gene length for a given level of DE.”
BackgroundMethodInfluentialLi, H., Lovci, M., Kwon, Y., Rosenfeld, M. G., Fu, X., & Yeo, G. W.-M. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0807121105 · Cited by 94 · cited 2× · see citing works
“0 arrays (GSE5350, samples AFX_1_ [A-B] [1-5]; see [Additional file 2: Supplemental Sections S1.”
BackgroundMethodGenomeGraphs: integrated genomic data visualization with R
Durinck, S., Bullard, J. H., Spellman, P. T., & Dudoit, S. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-2 · Cited by 83 · cited 4× · see citing works
“Software In order to facilitate analysis and visualization of mRNA-Seq data, we developed two R/Bioconductor software packages, Genominator and GenomeGraphs [20].”
MethodInfluentialSelf-citationLu, J., Tomfohr, J. K., & Kepler, T. B. (2005) · BMC Bioinformatics · doi:10.1186/1471-2105-6-165 · Cited by 79 · see citing works
Lee, A., Hansen, K. D., Bullard, J. H., Dudoit, S., & Sherlock, G. J. (2008) · PLoS Genetics · doi:10.1371/journal.pgen.1000299 · Cited by 34 · cited 3× · see citing works
“The Poisson distribution has been shown to provide a good fit to the distribution of gene-level counts across replicate lanes, after normalization by total lane counts [4,6]; our experience with both the MAQC data and”
Self-citation
Flagged first, then by citations
1 of the 23 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Taub MA Analysis of high-throughput biological data: some statistical problems in RNA-seq and mouse genotyping PhD thesis 2009 Department of Statistics, UC Berkeley
- Irizarry RA Hobbs B Collin F Beazer-Barclay YD Antonellis KJ Scherf U Speed TP Exploration, normalization, and summaries of high density oligonucleotide array probe level data Biostatistics 2003 4 1465-4644 (Print) 249 64 10.1093/biostatistics/4.2.249 12925520 · DOI · Paper page
Cited by
Sorted by citations
RSEM: accurate transcript quantification from RNA-Seq data with or without a reference genome
Li, B., & Dewey, C. N. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-323 · Cited by 22,179 · see citing works
Differential expression analysis for sequence count data
Anders, S., & Huber, W. (2010) · Genome Biology · doi:10.1186/gb-2010-11-10-r106 · Cited by 15,581 · see citing works
Trapnell, C., Williams, B. A., Pertea, G., Mortazavi, A., Kwan, G., van Baren, M. J., Salzberg, S. L., Wold, B. J., & Pachter, L. (2010) · Nature Biotechnology · doi:10.1038/nbt.1621 · Cited by 15,556 · see citing works
Haas, B. J., Papanicolaou, A., Yassour, M., Grabherr, M., Blood, P. D., Bowden, J., Couger, M. B., Eccles, D., Li, B., Lieber, M., et al. (2013) · Nature Protocols · doi:10.1038/nprot.2013.084 · Cited by 8,464 · see citing works
McCarthy, D. J., Chen, Y., & Smyth, G. K. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks042 · Cited by 5,195 · see citing works
Waste Not, Want Not: Why Rarefying Microbiome Data Is Inadmissible
McMurdie, P. J., & Holmes, S. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003531 · Cited by 2,891 · see citing works
A survey of best practices for RNA-seq data analysis
Conesa, A., Madrigal, P., Tarazona, S., Gomez-Cabrero, D., Cervera, A., McPherson, A., Szcześniak, M. W., Gaffney, D. J., Elo, L. L., Zhang, X., et al. (2016) · Genome Biology · doi:10.1186/s13059-016-0881-8 · Cited by 2,794 · see citing works
“Normalization methods that take this into account are TMM [47], DESeq [48], PoissonSeq [49] and UpperQuartile [45], which ignore highly variable and/or highly expressed features.”
MethodEarly allopolyploid evolution in the post-Neolithic Brassica napus oilseed genome
Chalhoub, B., Denœud, F., Liu, S., Parkin, I. A. P., Tang, H., Wang, X., Chiquet, J., Belcram, H., Tong, C., Samans, B., et al. (2014) · Science · doi:10.1126/science.1253435 · Cited by 2,464 · see citing works
Differential abundance analysis for microbial marker-gene surveys
Paulson, J. N., Stine, O. C., Bravo, H. C., & Pop, M. (2013) · Nature Methods · doi:10.1038/nmeth.2658 · Cited by 2,437 · see citing works
McCarthy, D. J., Campbell, K. R., Lun, A. T. L., & Wills, Q. F. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btw777 · Cited by 1,852 · see citing works
Risso (2014) · doi:10.1038/nbt.2931
Self-citationNormalization and microbial differential abundance strategies depend upon data characteristics
Weiss, S. J., Xu, Z. Z., Peddada, S. D., Amir, A., Bittinger, K. L., González, A., Lozupone, C., Zaneveld, J., Vázquez‐Baeza, Y., Birmingham, A., et al. (2017) · Microbiome · doi:10.1186/s40168-017-0237-y · Cited by 2,121 · see citing works
“(none, proportion, rarefying without replacement as in the hypergeometric model [75], CSS [4], logUQ [28], DESeqVS [30], and edgeR-TMM [27]) and the beta”
BackgroundMethodInfluentialReimand, J., Isserlin, R., Voisin, V., Kucera, M., Tannus-Lopes, C., Rostamianfar, A., Wadi, L., Meyer, M., Wong, J., Xu, C., et al. (2019) · Nature Protocols · doi:10.1038/s41596-018-0103-9 · Cited by 1,970 · see citing works
RNA sequencing: the teenage years
Stark, R., Grzelak, M., & Hadfield, J. (2019) · Nature Reviews Genetics · doi:10.1038/s41576-019-0150-2 · Cited by 1,906 · see citing works
Improving RNA-Seq expression estimates by correcting for fragment bias
Roberts, A., Trapnell, C., Donaghey, J., Rinn, J. L., & Pachter, L. (2011) · Genome Biology · doi:10.1186/gb-2011-12-3-r22 · Cited by 1,725 · see citing works
De novo assembly of bacterial transcriptomes from RNA-seq data
Tjaden, B. C. (2015) · Genome biology · doi:10.1186/s13059-014-0572-2 · Cited by 1,695 · see citing works
Chong, J., Liu, P., Zhou, G., & Xia, J. (2020) · Nature Protocols · doi:10.1038/s41596-019-0264-1 · Cited by 1,670 · see citing works
Full-length mRNA-Seq from single-cell levels of RNA and individual circulating tumor cells
Ramsköld, D., Luo, S., Wang, Y.-C., Li, R., Deng, Q., Faridani, O. R., Daniels, G. A., Khrebtukova, I., Loring, J. F., Laurent, L. C., et al. (2012) · Nature Biotechnology · doi:10.1038/nbt.2282 · Cited by 1,609 · see citing works
Differential expression in RNA-seq: A matter of depth
Tarazona, S., García-Alcalde, F., Dopazo, J., Ferrer, A., & Conesa, A. (2011) · Genome Research · doi:10.1101/gr.124321.111 · Cited by 1,632 · see citing works
Sequencing depth and coverage: key considerations in genomic analyses
Sims, D., Sudbery, I. M., Ilott, N. E., Heger, A., & Ponting, C. P. (2014) · Nature Reviews Genetics · doi:10.1038/nrg3642 · Cited by 1,403 · see citing works
Varet, H., Brillet-Guéguen, L., Coppée, J., & Dillies, M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157022 · Cited by 1,352 · see citing works
“Normalization is another key issue when looking for differentially expressed (DE) genes [10].”
BackgroundEBSeq: an empirical Bayes hierarchical model for inference in RNA-seq experiments
Leng, N., Dawson, J. A., Thomson, J. A., Ruotti, V., Rissman, A. I., Smits, B. M. G., Haag, J. D., Gould, M. N., Stewart, R. M., & Kendziorski, C. M. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt087 · Cited by 1,304 · see citing works
Dillies, M., Rau, A., Aubert, J., Hennequet‐Antier, C., Jeanmougin, M., Servant, N., Keime, C., Marot, G., Castel, D., Estellé, J., et al. (2012) · Briefings in Bioinformatics · doi:10.1093/bib/bbs046 · Cited by 1,281 · see citing works
Ingolia, N. T., Brar, G. A., Rouskin, S., McGeachy, A. M., & Weissman, J. S. (2012) · Nature Protocols · doi:10.1038/nprot.2012.086 · Cited by 1,240 · see citing works
Count-based differential expression analysis of RNA sequencing data using R and Bioconductor
Anders, S., McCarthy, D. J., Chen, Y., Okoniewski, M. J., Smyth, G. K., Huber, W. K., & Robinson, M. D. (2013) · Nature Protocols · doi:10.1038/nprot.2013.099 · Cited by 1,129 · see citing works
Computational methods for transcriptome annotation and quantification using RNA-seq
Garber, M., Grabherr, M. G., Guttman, M., & Trapnell, C. (2011) · Nature Methods · doi:10.1038/nmeth.1613 · Cited by 1,042 · see citing works
Guidelines for the use of flow cytometry and cell sorting in immunological studies (second edition)
Cossarizza, A., Chang, H., Radbruch, A., Acs, A., Adam, D., Adam‐Klages, S., Agace, W. W., Aghaeepour, N., Akdiş, M., Allez, M., et al. (2019) · European Journal of Immunology · doi:10.1002/eji.201970107 · Cited by 982 · see citing works
Commensal Microbiota Promote Lung Cancer Development via γδ T Cells
Jin, C., Lagoudas, G. K., Zhao, C., Bullman, S., Bhutkar, A., Hu, B., Ameh, S., Sandel, D., Liang, X. S., Mazzilli, S., et al. (2019) · Cell · doi:10.1016/j.cell.2018.12.040 · Cited by 964 · see citing works
A comparison of methods for differential expression analysis of RNA-seq data
Soneson, C., & Delorenzi, M. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-91 · Cited by 887 · see citing works
Schurch, N. J., Schofield, P., Gierliński, M., Cole, C., Sherstnev, A., Singh, V., Wrobel, N., Gharbi, K., Simpson, G. G., Owen-Hughes, T., et al. (2016) · RNA · doi:10.1261/rna.053959.115 · Cited by 966 · see citing works
“The majority of studies based on analysis of experimental RNA-seq data rely on data from experiments with fewer than five replicates per condition (Marioni et al. 2008; Bullard et al. 2010; Kvam et al. 2012; Li et al. 2012; Busby et al. 2013; Dillies et al. 2013; Rapaport et al. 2013; SEQC/MAQC-III Consortium 2014; So…”
MethodInfluentialMonaco, G., Lee, B., Xu, W., Mustafah, S., Hwang, Y. Y., Carré, C., Burdin, N., Visan, L., Ceccarelli, M., Poidinger, M., et al. (2019) · Cell Reports · doi:10.1016/j.celrep.2019.01.041 · Cited by 958 · see citing works
Kukurba, K. R., & Montgomery, S. B. (2015) · Cold Spring Harbor Protocols · doi:10.1101/pdb.top084970 · Cited by 924 · see citing works
GC-Content Normalization for RNA-Seq Data
Risso, D., Schwartz, K., Sherlock, G., & Dudoit, S. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-480 · Cited by 867 · see citing works
Self-citationTrajectory-based differential expression analysis for single-cell sequencing data
den Berge, K. V., de Bézieux, H. R., Street, K., Saelens, W., Cannoodt, R., Saeys, Y., Dudoit, S., & Clement, L. (2020) · Nature Communications · doi:10.1038/s41467-020-14766-3 · Cited by 796 · see citing works
Self-citationComprehensive evaluation of differential gene expression analysis methods for RNA-seq data
Rapaport, F., Khanin, R., Liang, Y., Pirun, M., Krek, A., Zumbo, P., Mason, C. E., Socci, N. D., & Betel, D. (2013) · Genome Biology · doi:10.1186/gb-2013-14-9-r95 · Cited by 733 · see citing works
RobiNA: a user-friendly, integrated software solution for RNA-Seq-based transcriptomics
Lohse, M., Bolger, A., Nagel, A., Fernie, A. R., Lunn, J. E., Stitt, M., & Usadel, B. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks540 · Cited by 778 · see citing works
Data quality aware analysis of differential expression in RNA-seq with NOISeq R/Bioc package
Tarazona, S., Furió‐Tarí, P., Turrà, D., Pietro, A. D., Nueda, M. J., Ferrer, A., & Conesa, A. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv711 · Cited by 757 · see citing works
“The NOISeq package includes three commonly used normalization methods: RPKM (4), Upper Quartile (15) and TMM (14).”
MethodBiases in Illumina transcriptome sequencing caused by random hexamer priming
Hansen, K. D., Brenner, S. E., & Dudoit, S. (2010) · Nucleic Acids Research · doi:10.1093/nar/gkq224 · Cited by 713 · see citing works
Self-citationbaySeq: Empirical Bayesian methods for identifying differential expression in sequence count data
Hardcastle, T. J., & Kelly, K. A. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-422 · Cited by 755 · see citing works
Revealing the vectors of cellular identity with single-cell genomics
Wagner, A., Regev, A., & Yosef, N. (2016) · Nature Biotechnology · doi:10.1038/nbt.3711 · Cited by 632 · see citing works
A general and flexible method for signal extraction from single-cell RNA-seq data
Risso, D., Perraudeau, F., Gribkova, S., Dudoit, S., & Vert, J.-P. (2018) · Nature Communications · doi:10.1038/s41467-017-02554-5 · Cited by 718 · see citing works
“• Full-quantile normalization (FQ) [54].”
Self-citationFrom RNA-seq reads to differential expression results
Oshlack, A., Robinson, M. D., & Young, M. D. (2010) · Genome biology · doi:10.1186/gb-2010-11-12-220 · Cited by 697 · see citing works
Quantitative assessment of single-cell RNA-sequencing methods
Wu, A. R., Neff, N. F., Kalisky, T., Dalerba, P., Treutlein, B., Rothenberg, M. E., Mburu, F. M., Mantalas, G. L., Sim, S., Clarke, M. F., et al. (2014) · Nature Methods · doi:10.1038/nmeth.2694 · Cited by 696 · see citing works
Removing technical variability in RNA-seq data using conditional quantile normalization
Hansen, K. D., Irizarry, R. A., & WU, Z. (2012) · Biostatistics · doi:10.1093/biostatistics/kxr054 · Cited by 607 · see citing works
Self-citationPatro, R., Mount, S. M., & Kingsford, C. (2014) · Nature Biotechnology · doi:10.1038/nbt.2862 · Cited by 586 · see citing works
A quantitative atlas of polyadenylation in five mammalsPreprint
Derti, A., Garrett-Engele, P. W., MacIsaac, K. D., Stevens, R. C., Sriram, S., Chen, R., Rohl, C. A., Johnson, J. M., & Babak, T. (2012) · Genome Research · doi:10.1101/gr.132563.111 · Cited by 642 · see citing works
Synthetic spike-in standards for RNA-seq experiments
Jiang, L., Schlesinger, F., Davis, C. A., Zhang, Y., Li, R., Salit, M., Gingeras, T. R., & Oliver, B. (2011) · Genome Research · doi:10.1101/gr.121095.111 · Cited by 653 · see citing works
Keap1 loss promotes Kras-driven lung cancer and results in dependence on glutaminolysis
Romero, R., Sayin, V. I., Davidson, S. M., Bauer, M. R., Singh, S. X., LeBoeuf, S. E., Karakousi, T. R., Ellis, D. C., Bhutkar, A., Sánchez-Rivera, F. J., et al. (2017) · Nature Medicine · doi:10.1038/nm.4407 · Cited by 652 · see citing works
TCC: an R package for comparing tag count data with robust normalization strategies
Sun, J., Nishiyama, T., Shimizu, K., & Kadota, K. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-219 · Cited by 613 · see citing works
RNA-Seq differential expression analysis: An extended review and a software tool
Silva, J. C., Domingues, D. S., & Lopes, F. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0190152 · Cited by 624 · see citing works
Computational analysis of bacterial RNA-Seq data
McClure, R., Balasubramanian, D., Sun, Y., Bobrovskyy, M., Sumby, P., Genco, C. A., Vanderpool, C. K., & Tjaden, B. C. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt444 · Cited by 584 · see citing works
Widespread macromolecular interaction perturbations in human genetic disorders
Sahni, N., Yi, S., Taipale, M., Fuxman Bass, J. I., Coulombe-Huntington, J., Yang, F., Peng, J., Weile, J., Karras, G. I., Wang, Y., et al. (2015) · Cell · doi:10.1016/j.cell.2015.04.013 · Cited by 594 · see citing works
Metabolic landscape of the tumor microenvironment at single cell resolution
Xiao, Z., Dai, Z., & Locasale, J. W. (2019) · Nature Communications · doi:10.1038/s41467-019-11738-0 · Cited by 524 · see citing works
Xia, Z., Donehower, L., Cooper, T., Neilson, J., Wheeler, D., Wagner, E., & Li, W. (2014) · Nature Communications · doi:10.1038/ncomms6274 · Cited by 509 · see citing works
TT-seq maps the human transient transcriptome
Schwalb, B., Michel, M., Zacher, B., Frühauf, K., Demel, C., Tresch, A., Gagneur, J., & Cramer, P. (2016) · Science · doi:10.1126/science.aad9841 · Cited by 548 · see citing works
The genomic landscape underlying phenotypic integrity in the face of gene flow in crows
Poelstra, J. W., Vijay, N., Bossu, C. M., Lantz, H., Ryll, B., Müller, I., Baglione, V., Unneberg, P., Wikelski, M. C., Grabherr, M. G., et al. (2014) · Science · doi:10.1126/science.1253226 · Cited by 542 · see citing works
svaseq: removing batch effects and other unwanted noise from sequencing data
Leek, J. T. (2014) · Nucleic Acids Research · doi:10.1093/nar/gku864 · Cited by 538 · see citing works
Microarrays, deep sequencing and the true measure of the transcriptome
Malone, J. H., & Oliver, B. (2011) · BMC Biology · doi:10.1186/1741-7007-9-34 · Cited by 546 · see citing works
Normalizing single-cell RNA sequencing data: challenges and opportunities
Vallejos, C. A., Risso, D., Scialdone, A., Dudoit, S., & Marioni, J. C. (2017) · Nature Methods · doi:10.1038/nmeth.4292 · Cited by 436 · see citing works
“However, these estimates can be dominated by a handful of highly expressed genes, which can bias downstream results [16, 22].”
BackgroundMethodInfluentialSelf-citationBenchmarking of cell type deconvolution pipelines for transcriptomics data
Avila Cobos, F., Alquicira-Hernandez, J., Powell, J. E., Mestdagh, P., & De Preter, K. (2020) · Nature Communications · doi:10.1038/s41467-020-19015-1 · Cited by 446 · see citing works
Miller, R., Wu, G., Deshpande, R. R., Vieler, A., Gärtner, K., Li, X., Moellering, E. R., Zäuner, S., Cornish, A. J., Liu, B., et al. (2010) · PLANT PHYSIOLOGY · doi:10.1104/pp.110.165159 · Cited by 511 · see citing works
Zhao, Y., Li, M., Konaté, M. M., Chen, L., Das, B., Karlovich, C., Williams, P. M., Evrard, Y. A., Doroshow, J. H., & McShane, L. M. (2021) · Journal of Translational Medicine · doi:10.1186/s12967-021-02936-w · Cited by 494 · see citing works
Hollern, D. P., Xu, N., Thennavan, A., Glodowski, C. R., García‐Recio, S., Mott, K. R., He, X., Garay, J. P., Carey-Ewend, K., Marron, D. L., et al. (2019) · Cell · doi:10.1016/j.cell.2019.10.028 · Cited by 470 · see citing works
Comparison of software packages for detecting differential expression in RNA-seq studies
Seyednasrollah, F., Laiho, A., & Elo, L. L. (2015) · Briefings in Bioinformatics · doi:10.1093/bib/bbt086 · Cited by 421 · see citing works
Klambauer, G., Schwarzbauer, K., Mayr, A., Clevert, D.-A., Mitterecker, A., Bodenhofer, U., & Hochreiter, S. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks003 · Cited by 440 · see citing works
Linear models enable powerful differential activity analysis in massively parallel reporter assays
Myint, L., Avramopoulos, D., Goff, L., & Hansen, K. (2017) · BMC Genomics · doi:10.1186/s12864-019-5556-x · Cited by 140 · see citing works
“On the log scale, low counts are more variable across replicates than high counts, at least partly due to inherent Poisson variation in the sequencing process [27, 28].”
BackgroundSelf-citationDesign and computational analysis of single-cell RNA-sequencing experiments
Bacher, R. L., & Kendziorski, C. M. (2016) · Genome biology · doi:10.1186/s13059-016-0927-y · Cited by 437 · see citing works
“Given the importance of normalization, it is not surprising that many normalization methods are available for bulk RNA-seq experiments [40 – 46], and these methods have been used in the majority of reported scRNA-seq experiments to date.”
MethodLi, J., & Tibshirani, R. J. (2011) · Statistical Methods in Medical Research · doi:10.1177/0962280211428386 · Cited by 429 · see citing works
GFOLD: a generalized fold change for ranking differentially expressed genes from RNA-seq data
Feng, J., Meyer, C. A., Wang, Q., Liu, J. S., Liu, X. S., & Zhang, Y. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts515 · Cited by 426 · see citing works
Ribosome Footprint Profiling of Translation throughout the Genome
Ingolia, N. T. (2016) · Cell · doi:10.1016/j.cell.2016.02.066 · Cited by 404 · see citing works
“Statistical frameworks for read count analysis typically account for this library size factor but rely on the assumption that most genes show similar expression between different samples (Bullard et al., 2010).”
BackgroundNormalization, testing, and false discovery rate estimation for RNA-sequencing data
Li, J., Witten, D. M., Johnstone, I. M., & Tibshirani, R. (2012) · Biostatistics · doi:10.1093/biostatistics/kxr031 · Cited by 369 · see citing works
Analysis of microbial compositions: a review of normalization and differential abundance analysis
Lin, H., & Peddada, S. D. (2020) · npj Biofilms and Microbiomes · doi:10.1038/s41522-020-00160-w · Cited by 341 · see citing works
George, J., Walter, V., Peifer, M., Alexandrov, L. B., Seidel, D., Leenders, F., Maas, L., Müller, C., Dahmen, I., Delhomme, T. M., et al. (2018) · Nature Communications · doi:10.1038/s41467-018-03099-x · Cited by 379 · see citing works
Selecting between-sample RNA-Seq normalization methods from the perspective of their assumptions
Evans, C., Hardin, J., & Stoebel, D. M. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx008 · Cited by 368 · see citing works
“Normalization for qRT-PCR often relies on housekeeping genes [10, 37, 38].”
BackgroundMethodInfluentialMethods for normalizing microbiome data: An ecological perspective
McKnight, D. T., Huerlimann, R., Bower, D. S., Schwarzkopf, L., Alford, R. A., & Zenger, K. R. (2018) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13115 · Cited by 373 · see citing works
<i>Polyester</i> : simulating RNA-seq datasets with differential transcript expression
Frazee, A. C., Jaffe, A. E., Langmead, B., & Leek, J. T. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv272 · Cited by 313 · see citing works
Transcriptomics in the RNA-seq era
McGettigan, P. (2013) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2012.12.008 · Cited by 360 · see citing works
A Comparative Study of Techniques for Differential Expression Analysis on RNA-Seq Data
Zhang, Z. H., Jhaveri, D. J., Marshall, V. M., Bauer, D. C., Edson, J., Narayanan, R. K., Robinson, G. J., Lundberg, A. E., Bartlett, P. F., Wray, N. R., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0103207 · Cited by 357 · see citing works
Hugerth, L. W., & Andersson, A. F. (2017) · Frontiers in Microbiology · doi:10.3389/fmicb.2017.01561 · Cited by 359 · see citing works
“A similar approach is using not the total count of reads for normalization, but a fixed percentile of them (Bullard et al., 2010), which should be less sensitive to events such as blooms.”
MethodInfluentialNext maSigPro: updating maSigPro bioconductor package for RNA-seq time series
Nueda, M. J., Tarazona, S., & Conesa, A. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu333 · Cited by 348 · see citing works
Comparative analysis of RNA-Seq alignment algorithms and the RNA-Seq unified mapper (RUM)
Grant, G. R., Farkas, M. H., Pizarro, A., Lahens, N. F., Schug, J., Brunk, B. P., Stoeckert, C. J., Hogenesch, J. B., & Pierce, E. A. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr427 · Cited by 336 · see citing works
Li, Y., Ge, X., Peng, F., Li, W., & Li, J. J. (2022) · Genome Biology · doi:10.1186/s13059-022-02648-4 · Cited by 303 · see citing works
Nookaew, I., Papini, M., Pornputtapong, N., Scalcinati, G., Fagerberg, L., Uhlén, M., & Nielsen, J. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks804 · Cited by 309 · see citing works
Lund, S. P., Nettleton, D. S., McCarthy, D. J., & Smyth, G. K. (2012) · Statistical Applications in Genetics and Molecular Biology · doi:10.1515/1544-6115.1826 · Cited by 322 · see citing works
Pérez-Cobas, A. E., Gomez-Valero, L., & Buchrieser, C. (2020) · Microbial Genomics · doi:10.1099/mgen.0.000409 · Cited by 314 · see citing works
“A simple scaling method, the log upper quartile (logUQ), basically scales each sample by the 75th percentile of the distribution of counts before log transformation [215].”
MethodCloud-scale RNA-sequencing differential expression analysis with Myrna
Langmead, B., Hansen, K. D., & Leek, J. T. (2010) · Genome biology · doi:10.1186/gb-2010-11-8-r83 · Cited by 313 · see citing works
Self-citationInnate-like functions of natural killer T cell subsets result from highly divergent gene programs
Engel, I., Seumois, G., Chávez, L., Samaniego‐Castruita, D., White, B. M., Chawla, A., Mock, D. C., Vijayanand, P., & Kronenberg, M. (2016) · Nature Immunology · doi:10.1038/ni.3437 · Cited by 311 · see citing works
TCGA Workflow: Analyze cancer genomics and epigenomics data using Bioconductor packages
Silva, T. C., Colaprico, A., Olsen, C., D’Angelo, F., Bontempi, G., Ceccarelli, M., & Noushmehr, H. (2016) · F1000Research · doi:10.12688/f1000research.8923.1 · Cited by 295 · see citing works
RNA-seq: technical variability and sampling
McIntyre, L. M., Lopiano, K. K., Morse, A. M., Amin, V., Oberg, A. L., Young, L. J., & Nuzhdin, S. V. (2011) · BMC Genomics · doi:10.1186/1471-2164-12-293 · Cited by 285 · see citing works
Evolutionary changes in promoter and enhancer activity during human corticogenesis
Reilly, S. K., Yin, J., Ayoub, A. E., Emera, D., Leng, J., Cotney, J., Sarro, R., Rakic, P., & Noonan, J. P. (2015) · Science · doi:10.1126/science.1260943 · Cited by 297 · see citing works
Power analysis and sample size estimation for RNA-Seq differential expression
Ching, T., Huang, S., & Garmire, L. X. (2014) · RNA · doi:10.1261/rna.046011.114 · Cited by 263 · see citing works
Twine, N. A., Janitz, K., Wilkins, M. R., & Janitz, M. (2011) · PLoS ONE · doi:10.1371/journal.pone.0016266 · Cited by 295 · see citing works
Resolving host–pathogen interactions by dual RNA-seq
Westermann, A. J., Barquist, L., & Vogel, J. (2017) · PLoS Pathogens · doi:10.1371/journal.ppat.1006033 · Cited by 286 · see citing works
“The most commonly used (and best performing [47]) RNA-seq normalization techniques address this problem by attempting to scale read counts by a factor derived from a set of putatively invariant genes identified through excluding genes with extreme differences in expression [48], the use of robust median statistics [49…”
MethodRNA sequencing reveals two major classes of gene expression levels in metazoan cells
Hebenstreit, D., Fang, M., Gu, M., Charoensawan, V., van Oudenaarden, A., & Teichmann, S. A. (2011) · Molecular Systems Biology · doi:10.1038/msb.2011.28 · Cited by 289 · see citing works
Da Silva, H. E., Teterina, A., Comelli, E. M., Taibi, A., Arendt, B. M., Fischer, S. E., Lou, W., & Allard, J. P. (2018) · Scientific Reports · doi:10.1038/s41598-018-19753-9 · Cited by 283 · see citing works
A comparison of statistical methods for detecting differentially expressed genes from RNA‐seq data
Kvam, V. M., Liu, P., & Si, Y. (2012) · American Journal of Botany · doi:10.3732/ajb.1100340 · Cited by 281 · see citing works
Advanced Applications of RNA Sequencing and Challenges
Han, Y., Gao, S., Muegge, K., Zhang, W., & Zhou, B. (2015) · Bioinformatics and Biology Insights · doi:10.4137/bbi.s28991 · Cited by 277 · see citing works
Zeng, A., Li, H., Guo, L., Gao, X., McKinney, S., Wang, Y., Yu, Z., Park, J., Semerad, C. L., Ross, E. J., et al. (2018) · Cell · doi:10.1016/j.cell.2018.05.006 · Cited by 280 · see citing works
Dietary Fat, but Not Protein or Carbohydrate, Regulates Energy Intake and Causes Adiposity in Mice
Hu, S., Wang, L., Yang, D., Li, L., Togo, J., Wu, Y., Liu, Q., Li, B., Li, M., Wang, G., et al. (2018) · Cell Metabolism · doi:10.1016/j.cmet.2018.06.010 · Cited by 277 · see citing works
RD26 mediates crosstalk between drought and brassinosteroid signalling pathways
Ye, H., Liu, S., Tang, B., Chen, J., Xie, Z., Nolan, T. M., Jiang, H., Guo, H., Lin, H.-Y., Li, L., et al. (2017) · Nature Communications · doi:10.1038/ncomms14573 · Cited by 272 · see citing works
“The 0.75 quantile of reads from each sample was used as the normalization factor 65 .”
MethodInfluential
- Authors
- Bullard, J. H., Purdom, E., Hansen, K. D., & Dudoit, S.
- Affiliations
- University of California, Berkeley (US)
- Corresponding author
- Bullard, James H
- Venue
- BMC Bioinformatics
- Published
- 2010-02-18
- Volume
- 11
- Issue
- 1
- Pages
- 94-94
- ISSN
- 1471-2105 (ISSN-L)
- PMC
- PMC2838869
- OpenAlex
- W2156631105
- Cited by
- 1,680
- References
- 23
- Citations by year
- 2026: 29 · 2025: 51 · 2024: 69 · 2023: 68 · 2022: 92 · 2021: 132 · 2020: 127 · 2019: 123 · 2018: 123 · 2017: 133 · 2016: 121 · 2015: 129 · 2014: 161 · 2013: 128 · 2012: 98
- FWCI
- 41.79
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Bioinformatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Collection of Biostatistics Research Archive (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- National Science Foundation
- National Institutes of Health
- National Human Genome Research Institute (U01HG004271)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Keywords
- mRNA-seq, read count normalization, differential gene expression, method comparison
- MeSH terms
- RNA, Messenger, Sequence Analysis, RNA, Computational Biology, Databases, Genetic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
