The DOI matches its Crossref record; 22 of the 23 references it cites were found in a registry or an index.
Evaluation of statistical methods for normalization and differential expression in mRNA-Seq experiments
Open accessAbstract
BACKGROUND: High-throughput sequencing technologies, such as the Illumina Genome Analyzer, are powerful new tools for investigating a wide range of biological and medical questions. Statistical and computational methods are key for drawing meaningful and accurate conclusions from the massive and complex datasets generated by the sequencers. We provide a detailed evaluation of statistical methods for normalization and differential expression (DE) analysis of Illumina transcriptome sequencing (mRNA-Seq) data. RESULTS: We compare statistical methods for detecting genes that are significantly DE between two types of biological samples and find that there are substantial differences in how the test statistics handle low-count genes. We evaluate how DE results are affected by features of the sequencing platform, such as, varying gene lengths, base-calling calibration method (with and without phi X control lane), and flow-cell/library preparation effects. We investigate the impact of the read count normalization method on DE results and show that the standard approach of scaling by total lane counts (e.g., RPKM) can bias estimates of DE. We propose more general quantile-based normalization procedures and demonstrate an improvement in DE detection. CONCLUSIONS: Our results have significant practical and methodological implications for the design and analysis of mRNA-Seq experiments. They highlight the importance of appropriate statistical methods for normalization and DE inference, to account for features of the sequencing platform that could impact the accuracy of results. They also reveal the need for further research in the development of statistical and computational methods for mRNA-Seq.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 4 Comparison of biological, library preparation, and flow-cell effects. Boxplots of estimated log-fold-changes for UHR vs. Brain biological effects (GLM 2 in [Additional file 2: Supplemental Table S4]), flow-cell effects adjusting for biology (GLM 4), library preparation effects within flow-cell (GLM 7). Estimates are presented for total-count (black) and upper-quartile (blue) normalization.](/paper-figures/7a218023bc1142f6aea616fd271add05/evaluation-of-statistical-methods-for-normalization-and-differential-expression-figure-4.jpg)
![Figure 6 Comparison of mRNA-Seq and microarray differential expression calls to qRT-PCR: ROC curves. Genes common to all three platforms and present for both qRT-PCR and sequencing (see [Additional file 2: Supplemental Section S6]). evaluated and declared DE if their qRT-PCR absolute log-ratio was (a) greater than 2 or (b) greater than 0.5; genes were declared non-DE if their absolute log-ratio was less than 0.2. The GLM-based likelihood ratio test was used for the sequencing data. Two normalization procedures are presented for mRNA-Seq: total-count (black) and upper-quartile (blue) normalization. Microarray data were normalized using RMA (gray). Note that we require a true positive to be differentially expressed in the same direction according to both mRNA-Seq and qRT-PCR (see Table 1 and Methods).](/paper-figures/e79890d44a604905bdf241f0104817db/evaluation-of-statistical-methods-for-normalization-and-differential-expression-figure-6.jpg)
![Figure 7 Comparison of mRNA-Seq and microarray differential expression measures to qRT-PCR. Difference scatterplots comparing the estimates of UHR/Brain expression log-ratio from qRT-PCR to those from (a) mRNA-Seq, using the standard total-count normalization, and (b) microarrays, using the standard RMA normalization. Shown are the genes shared between all three platforms, present in both Brain and UHR according to both mRNA-Seq and qRT-PCR (see [Additional file 2: Supplemental Section S1]), and having absolute qRT-PCR expression log-ratio less than 4. Horizontal lines in (a) represent the median UHR/Brain log-ratio for the sequencing data after the standard total-count normalization (black), POLR2A normalization (red), quantile normalization (yellow), upper-quartile normalization (blue); horizontal lines in (b) show the median UHR/Brain log-ratio for the microarray data after the standard RMA normalization (black) and POLR2A normalization (red).](/paper-figures/ffe14db2d64a46939165f0c6e8ca7bec/evaluation-of-statistical-methods-for-normalization-and-differential-expression-figure-7.jpg)
![Figure 8 Comparison of normalization procedures: Goodness-of-fit of Poisson model. The multiplicative Poisson model (GLM 1 in [Additional file 2: Supplemental Table S4]) is fit to the seven Brain lanes in the MAQC-2 experiment after (a) total-count, (b) POLR2A, (c) upper-quartile, and (d) quantile normalization. Goodness-of-fit statistics are computed and displayed in χ2 quantile-quantile plots. Genes with goodness-of-fit statistics in the top quantiles of the χ2-distribution are displayed using colored plotting symbols: red (1, 5]%, purple (.1, 1]%, gold [0, .1]%. Similar plots for UHR show the same patterns.](/paper-figures/d99eb5ce90ee4a9ebac8c9f438b4c283/evaluation-of-statistical-methods-for-normalization-and-differential-expression-figure-8.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These approaches rely on normalizing methods that are based on total or effective counts, and tend to perform poorly when samples have heterogeneous transcript distributions, that is, when highly and differentially expressed features can skew the count distribution [45, 46].”
From A survey of best practices for RNA-seq data analysis (2016) · cites this paper as [45]
Experimental — accuracy not yet measuredIn the citing paper
“While reproducibility in RNA-seq is usually high at the level of sequencing [1, 45], other steps such as RNA extraction and library preparation are noisier and may introduce biases in the data that can be minimized by adopting good experimental procedures (Box 2).”
From A survey of best practices for RNA-seq data analysis (2016) · cites this paper as [45]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Ultrafast and memory-efficient alignment of short DNA sequences to the human genome.
Langmead, B., Trapnell, C., Pop, M., & Salzberg, S. L. (2009) · Genome Biology · doi:10.1186/gb-2009-10-3-r25 · Cited by 21,849 · cited 1×
“We used Bowtie to map reads to the genome (GRCh37 assembly) [ 16 ].”
Method(2005) · Springer eBooks · doi:10.1007/0-387-27255-0_9 · Cited by 20,333
Mapping and quantifying mammalian transcriptomes by RNA-Seq
Mortazavi, A., Williams, B. A., McCue, K., Schaeffer, L., & Wold, B. (2008) · Nature Methods · doi:10.1038/nmeth.1226 · Cited by 13,696 · cited 6×
“These high-throughput sequencing technologies have already been applied to monitor genome-wide transcription levels (mRNA-Seq), DNA-protein interactions (ChIP-Seq), chromatin structure, and DNA methylation [1-9].”
BackgroundMethodInfluentialSmyth, G. K. (2004) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1027 · Cited by 11,829
Exploration, normalization, and summaries of high density oligonucleotide array probe level data
Irizarry, R. A. (2003) · Biostatistics · doi:10.1093/biostatistics/4.2.249 · Cited by 10,603
Base-Calling of Automated Sequencer Traces Using Phred. I. Accuracy AssessmentPreprint
Ewing, B., Hillier, L., Wendl, M. C., & Green, P. (1998) · Genome Research · doi:10.1101/gr.8.3.175 · Cited by 6,993
Base-Calling of Automated Sequencer Traces Using Phred. II. Error Probabilities
Ewing, B., & Green, P. (1998) · Genome Research · doi:10.1101/gr.8.3.186 · Cited by 5,439 · cited 1×
“quality scores for the other seven lanes [[15], Supplementary Information, p. 7]. Bustard can also be run using the auto-calibration method, which scores basecalls in a manner similar to the phred base-caller [ 17 ] and does not require a control lane per flow-cell.”
MethodAlternative isoform regulation in human tissue transcriptomes
Wang, E. T., Sandberg, R., Luo, S., Khrebtukova, I., Zhang, L., Mayr, C., Kingsmore, S. F., Schroth, G. P., & Burge, C. B. (2008) · Nature · doi:10.1038/nature07509 · Cited by 5,181
Accurate whole human genome sequencing using reversible terminator chemistry
Bentley, D. R., Balasubramanian, S., Swerdlow, H. P., Smith, G. P., Milton, J., Brown, C. G., Hall, K. P., Evers, D. J., Barnes, C. L., Bignell, H. R., et al. (2008) · Nature · doi:10.1038/nature07517 · Cited by 3,684 · cited 1×
“quality scores for the other seven lanes [[15], Supplementary Information, p.”
BackgroundRNA-seq: An assessment of technical reproducibility and comparison with gene expression arrays
Marioni, J. C., Mason, C. E., Mane, S. M., Stephens, M., & Gilad, Y. (2008) · Genome Research · doi:10.1101/gr.079558.108 · Cited by 2,771 · cited 7×
“These high-throughput sequencing technologies have already been applied to monitor genome-wide transcription levels (mRNA-Seq), DNA-protein interactions (ChIP-Seq), chromatin structure, and DNA methylation [1-9].”
BackgroundMethodResultInfluential
Research works printed but not linked: 4
- Illumina:Sequencing Analysis Software User Guide For Pipeline Version 1.3 and CASAVA Version 1.0 T. Illumina, Inc.; 2008. [Part # 1005359 Rev. A] [ http://icom.illumina.com/icom/software.ilmn?id=277 ] [Part # 1005359 Rev. A]
- Illumina:Preparing Samples for Sequencing mRNA. Ilumina, Inc.; 2009. [Part # 1004898 Rev. A] [ http://icom.illumina.com/icom/software.ilmn?id=277 ] [Part # 1004898 Rev. A]
- Taub MA: Analysis of high-throughput biological data: some statistical problems in RNA-seq and mouse genotyping. PhD thesis. Department of Statistics, UC Berkeley; 2009.
- Irizarry RA, Hobbs B, Collin F, Beazer-Barclay YD, Antonellis KJ, Scherf U, Speed TP: Exploration, normalization, and summaries of high density oligonucleotide array probe level data. Biostatistics 2003, 4(1465–4644 (Print)):249–64. 10.1093/biostatistics/4.2.249 · DOI · Paper page
Cited by
Sorted by citations
Items 201–300 of 999
Positional distribution of transcription factor binding sites in Arabidopsis thaliana
Yu, C.-P., Lin, J.-J., & Li, W. (2016) · Scientific Reports · doi:10.1038/srep25164 · Cited by 122
Dissecting Cellular Heterogeneity Using Single-Cell RNA Sequencing.
Choi, Y. H., & Kim, J. K. (2019) · Europe PMC (PubMed Central) · doi:10.14348/molcells.2019.2446 · Cited by 126
“…based on library size (e.g., reads per million (RPM) or transcripts per kilobase million (TPM)(Li et al., 2010)), upper quantile values of counts (Bullard et al., 2010), or normalization factors (e.g., size factor of DESeq (Anders and Huber, 2010) or trimmed mean of M-value of edgeR (Robinson…”
BackgroundAn approach for normalization and quality control for NanoString RNA expression data
Bhattacharya, A., Hamilton, A. M., Furberg, H., Pietzak, E., Purdue, M. P., Troester, M. A., Hoadley, K. A., & Love, M. I. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa163 · Cited by 122
MicroRNAs shape circadian hepatic gene expression on a transcriptome-wide scale
Du, N., Arpat, A. B., Matos, M. D., & Gatfield, D. (2014) · eLife · doi:10.7554/elife.02510 · Cited by 123
Analysis and correction of compositional bias in sparse sequencing count data
Kumar, M. S. B., SLUD, E. V., Okrah, K., Hicks, S. C., Hannenhalli, S. S., & Bravo, H. C. (2018) · BMC Genomics · doi:10.1186/s12864-018-5160-5 · Cited by 123
“This fact allows one to exploit several widely used non- spike-in normalization approaches [15, 13, 16, 17], which approximate the aforementioned spike-in strategy by assuming that most features do not change on average across samples/conditions.”
MethodA powerful and flexible approach to the analysis of RNA sequence count data
Zhou, Y., Xia, K., & Wright, F. A. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr449 · Cited by 124
From bench to bedside: Single-cell analysis for cancer immunotherapy
Davis-Marcisak, E. F., Deshpande, A. M., Stein-O’Brien, G., Ho, W. J., Laheru, D. A., Jaffee, E. M., Fertig, E. J., & Kagohara, L. T. (2021) · Cancer Cell · doi:10.1016/j.ccell.2021.07.004 · Cited by 126
APAtrap: identification and quantification of alternative polyadenylation sites from RNA-seq data
Ye, C., Long, Y., Ji, G., Li, Q. Q., & Wu, X. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty029 · Cited by 117
Peixoto, L., Risso, D., Poplawski, S. G., Wimmer, M. E., Speed, T. P., Wood, M. A., & Abel, T. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv736 · Cited by 105
“Similar results were obtained using TMM, while FPKM is the worst performing normalization method as previously shown (2,3) (Supplementary Figure S1A).”
MethodResultInjury Activates Transient Olfactory Stem Cell States with Diverse Lineage Capacities
Gadye, L., Das, D., Sanchez, M. A., Street, K., Baudhuin, A., Wagner, A., Cole, M. B., Choi, Y. G., Yosef, N., Purdom, E. A., et al. (2017) · Cell stem cell · doi:10.1016/j.stem.2017.10.014 · Cited by 117
Self-citationProbabilistic error correction for RNA sequencing
Le, H.-S., Schulz, M. H., McCauley, B. S., Hinman, V. F., & Bar‐Joseph, Z. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt215 · Cited by 125
SOX17 enables immune evasion of early colorectal adenomas and cancers
Goto, N., Westcott, P. M. K., Goto, S., Imada, S., Taylor, M. S., Eng, G. M., Braverman, J., Deshpande, V., Jacks, T., Agudo, J., et al. (2024) · Nature · doi:10.1038/s41586-024-07135-3 · Cited by 122
The Impact of Normalization Methods on RNA-Seq Data Analysis
Zyprych-Walczak, J., Szabelska‐Beręsewicz, A., Handschuh, L., Górczak, K., Klamecka, K., Figlerowicz, M., & Siatkowski, I. (2015) · BioMed Research International · doi:10.1155/2015/621690 · Cited by 125
“Taking into account all of these data sets the conclusion is that the best results were obtained using DES, EBS, and PS normalization methods, and the methods which generated the highest bias and variance values were TMM and UQ.”
BackgroundMethodInfluentialHernández‐Oñate, M. Á., Naranjo, E. U. E., Mendoza‐Mendoza, A., Stewart, A., & Herrera‐Estrella, A. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1209396109 · Cited by 115
Lawless, N., Foroushani, A. B. K., McCabe, M. S., O’Farrelly, C., & Lynn, D. J. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057543 · Cited by 115
An expanded landscape of human long noncoding RNA
Jiang, S., Cheng, S., Ren, L., Wang, Q., Kang, Y.-J., Ding, Y., Hou, M. L., Yang, X., Lin, Y., Liang, N., et al. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz621 · Cited by 116
Concepcion, C. P., Ma, S., LaFave, L. M., Bhutkar, A. J., Liu, M., DeAngelo, L. P., Kim, J. Y., Priore, I. S. D., Schoenfeld, A. J., Miller, M., et al. (2021) · Cancer Discovery · doi:10.1158/2159-8290.cd-21-0248 · Cited by 119
Phylogenetic ANOVA: The Expression Variance and Evolution Model for Quantitative Trait Evolution
Rohlfs, R. V., & Nielsen, R. (2015) · Systematic Biology · doi:10.1093/sysbio/syv042 · Cited by 116
“Expression levels were normalized based on each individual, transcript length, GC content, and species (Bullard et al. 2010; Pickrell et al. 2010; Perry et al. 2012), as is appropriate for comparative analysis so that genes are considered equitably in relation to each other (Dunn et al.”
BackgroundMethodTranscriptome dynamics of a broad host-range cyanophage and its hosts
Doron, S., Fedida, A., Hernández‐Prieto, M. A., Sabehi, G., Karunker, I., Stazic, D., Feingersch, R., Steglich, C., Futschik, M. E., Lindell, D., et al. (2015) · The ISME Journal · doi:10.1038/ismej.2015.210 · Cited by 114
“Normalization with a house-keeping gene has been shown to perform well for RNA-seq read counts (Bullard et al., 2010), and the rnpB gene was used for this purpose as the transcript levels of this gene were stable over the course of infection (Supplementary Figure S1), as has been shown previously…”
Resultquantro: a data-driven approach to guide the choice of an appropriate normalization method
Hicks, S. C., & Irizarry, R. A. (2015) · Genome biology · doi:10.1186/s13059-015-0679-0 · Cited by 117
“Quantile normalization was originally developed for gene expression microarrays [1, 2] but today it is applied in a wide-range of data types, including genotyping arrays [16, 17], RNA-Sequencing (RNA-Seq) [18–20], DNA methylation [21], ChIP-Sequencing [22, 23] and brain imaging [24–26].”
Tang, H. M., Liu, S., Hill‐Skinner, S., Wu, W., Reed, D., Yeh, C., Nettleton, D. S., & Schnable, P. S. (2013) · The Plant Journal · doi:10.1111/tpj.12394 · Cited by 110
Katayama, S., Töhönen, V., Linnarsson, S., & Kere, J. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt511 · Cited by 111
A normalization strategy for comparing tag count data
Kadota, K., Nishiyama, T., & Shimizu, K. (2012) · Algorithms for Molecular Biology · doi:10.1186/1748-7188-7-5 · Cited by 108
Rokyta, D. R., Wray, K. P., & Margres, M. J. (2013) · BMC Genomics · doi:10.1186/1471-2164-14-394 · Cited by 108
Johnson, K. A., & Krishnan, A. (2022) · Genome biology · doi:10.1186/s13059-021-02568-9 · Cited by 111
“studies have also noted that using RPKM does not necessarily take away the length bias in gene expression and can be unduly influenced by relatively few transcripts [13, 27].”
BackgroundGraphical Models via Univariate Exponential Family Distributions.
Yang, E., Ravikumar, P., Allen, G. I., & Liu, Z. (2015) · PubMed · Cited by 105
Evaluation of normalization methods in mammalian microRNA-Seq data
Garmire, L. X., & Subramaniam, S. (2012) · RNA · doi:10.1261/rna.030916.111 · Cited by 108
Han, K., Tjaden, B. C., & Lory, S. J. (2016) · Nature Microbiology · doi:10.1038/nmicrobiol.2016.239 · Cited by 106
Tabibian, J. H., Trussoni, C. E., O’Hara, S. P., Splinter, P. L., Heimbach, J. K., & LaRusso, N. F. (2014) · Laboratory Investigation · doi:10.1038/labinvest.2014.94 · Cited by 105
Global Analysis of Fission Yeast Mating Genes Reveals New Autophagy Factors
Sun, L., Li, M., Suo, F., Liu, X., Shen, E.-Z., Yang, B., Dong, M., He, W., & Du, L. (2013) · PLoS Genetics · doi:10.1371/journal.pgen.1003715 · Cited by 106
Geis-Asteggiante, L., Belew, A. T., Clements, V. K., Edwards, N. J., Ostrand-Rosenberg, S., El-Sayed, N. M., & Fenselau, C. (2018) · Journal of Proteome Research · doi:10.1021/acs.jproteome.7b00646 · Cited by 104
Type 2 innate lymphoid cells treat and prevent acute gastrointestinal graft-versus-host disease
Bruce, D., Stefanski, H. E., Vincent, B. G., Dant, T. A., Reisdorf, S., Bommiasamy, H., Serody, D. A., Wilson, J. E., McKinnon, K. P., Shlomchik, W. D., et al. (2017) · Journal of Clinical Investigation · doi:10.1172/jci91816 · Cited by 106
Nicolle, R., Ayadi, M., Gomez‐Brouchet, A., Armenoult, L., Banneau, G., Elarouci, N., Tallegas, M., Decouvelaere, A., Aubert, S., Rédiní, F., et al. (2019) · Nature Communications · doi:10.1038/s41467-019-12525-7 · Cited by 107
Nersisyan, S. A., Shkurnikov, M. Y., Turchinovich, A. A., Knyazev, E. N., & Tonevitsky, A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235987 · Cited by 103
“Fragments per kilobase of transcript per million mapped reads upper quartile (FPKM-UQ) scale [41] was used to quantify mRNA expression, while reads normalized by upper quartile (R-UQ) were used as an isomiR expression scale, all values were log2-transformed.”
MethodMolecular Features of Cancers Exhibiting Exceptional Responses to Treatment
Wheeler, D. A., Takebe, N., Hinoue, T., Hoadley, K. A., Cardenas, M., Hamilton, A. M., Laird, P. W., Wang, L., Johnson, A., Dewal, N., et al. (2020) · Cancer Cell · doi:10.1016/j.ccell.2020.10.015 · Cited by 103
“Data Analysis NanoString gene expression values were normalized with the method of Bhattacharya et al. (2020) with upper-quartile normalization (Bullard et al., 2010) andRemoveUnwanted Variation using Residuals (RUVr) using the RUVr function from theRUVSeqBioconductor package(Risso et al., 2014).”
MethodFrancis, W. R., Christianson, L. M., Kiko, R., Powers, M. L., Shaner, N. C., & Haddock, S. H. D. (2013) · BMC Genomics · doi:10.1186/1471-2164-14-167 · Cited by 101
Buschmann, D., Haberberger, A., Kirchner, B., Spornraft, M., Riedmaier, I., Schelling, G., & Pfaffl, M. W. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw545 · Cited by 102
“Although early RNA-Seq reported a good fit to a Poisson distribution for single sample sequencings and technical replicates (30,196), studies with biological replicates are extremely likely to show variances greater than the mean for many genes (204).”
BackgroundIsolation of Specific Neurons from C. elegans Larvae for Gene Expression Profiling
Spencer, W. C., McWhirter, R. D., Miller‐Fleming, T. W., Strasbourger, P., Thompson, O. A., Hillier, L., Waterston, R., & Miller, D. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112102 · Cited by 96
Connor, C., Dincer, A., Straubhaar, J., Galler, J. R., Houston, I. B., & Akbarian, S. (2012) · Schizophrenia Research · doi:10.1016/j.schres.2012.06.037 · Cited by 100
Normalization and noise reduction for single cell RNA-seq experiments
Ding, B., Zheng, L., Zhu, Y., Li, N., Jia, H., Ai, R., Wildberg, A., & Wang, W. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv122 · Cited by 99
“Contact: wei-wang@ucsd.edu Supplementary information: Supplementary data are available at Bioinformatics online.”
BackgroundInterpreting pathways to discover cancer driver genes with Moonlight
Colaprico, A., Olsen, C., Bailey, M. H., Odom, G., Terkelsen, T., Silva, T. C., Olsen, A. V., Cantini, L., Zinovyev, A. Y., Barillot, E., et al. (2020) · Nature Communications · doi:10.1038/s41467-019-13803-0 · Cited by 101
RNA‐seq Data: Challenges in and Recommendations for Experimental Design and Analysis
Williams, A. G., Thomas, S., Wyman, S. K., & Holloway, A. K. (2014) · Current Protocols in Human Genetics · doi:10.1002/0471142905.hg1113s83 · Cited by 93
Tumor-homing cytotoxic human induced neural stem cells for cancer therapy
Bagó, J. R., Okolie, O., Dumitru, R., Ewend, M. G., Parker, J. S., Werff, R. V., Underhill, T. M., Schmid, R. S., Miller, C. R., & Hingtgen, S. (2017) · Science Translational Medicine · doi:10.1126/scitranslmed.aah6510 · Cited by 102
“Before visualization, raw RSEM read counts for all RNA-seq samples were normalized to the overall upper quartile (53).”
MethodWang, M., Song, W., Chen, M., Wang, Q., Zhou, X., Xu, P., Krek, A., Yoon, Y., Ho, L., Orr, M. E., et al. (2022) · Molecular Neurodegeneration · doi:10.1186/s13024-022-00517-z · Cited by 104
“In the context of bulk RNA-seq, there are three most popular methods for robust scaling normalization, including 1) the trimmed mean of M-values (TMM) [77], which calculates scaling factors by trimming away genes with extreme fold changes between samples; 2) the upper-quartile (UQ) method [78], which uses persample up…”
MethodWilkins, K., Booher, N. J., Wang, L., & Bogdanove, A. J. (2015) · Frontiers in Plant Science · doi:10.3389/fpls.2015.00536 · Cited by 101
“Differentially expressed genes were identified using QuasiSeq (Lund et al., 2012) with the quartile normalization method (Bullard et al., 2010).”
MethodSequencing and characterization of the guppy (Poecilia reticulata) transcriptome
Fraser, B. A., Weadick, C. J., Janowitz, I., Rodd, F. H., & Hughes, K. A. (2011) · BMC Genomics · doi:10.1186/1471-2164-12-202 · Cited by 101
ATTED-II in 2014: Evaluation of Gene Coexpression in Agriculturally Important Plants
Obayashi, T., Okamura, Y., Ito, S., Tadaka, S., Aoki, Y., Shirota, M., & Kinoshita, K. (2013) · Plant and Cell Physiology · doi:10.1093/pcp/pct178 · Cited by 98
Transcriptome dynamics of Camellia sinensis in response to continuous salinity and drought stress
Zhang, Q., Cai, M., Yu, X., Wang, L., Guo, C., Ming, R., & Zhang, J. (2017) · Tree Genetics & Genomes · doi:10.1007/s11295-017-1161-9 · Cited by 95
“To compare expression levels of selected genes across all time points and reduce the bias in FPKM values due to highly expressed genes, the upper quartile normalization procedure (Bullard et al. 2010) was applied.”
Data analysis guidelines for single-cell RNA-seq in biomedical studies and clinical applications
Su, M., Pan, T., Chen, Q.-Z., Zhou, W., Gong, Y., Xu, G., Yan, H.-Y., Li, S., Shi, Q.-Z., Zhang, Y., et al. (2022) · Military Medical Research · doi:10.1186/s40779-022-00434-8 · Cited by 114
“Popular global-scaling methods for bulk RNA-seq include transcript per million (TPM) [85], upper quartile (UQ) normalization [86], trimmed mean of M values (TMM) normalization [87], and the DESeq normalization method [88], however, are not appropriate for scRNA-seq due the tendency for distortion…”
MethodGene Networks in Plant Biology: Approaches in Reconstruction and Analysis
Li, Y., Pearl, S. A., & Jackson, S. A. (2015) · Trends in Plant Science · doi:10.1016/j.tplants.2015.06.013 · Cited by 96
“For RNA-Seq data, DESeq [16] or edgeR [17] are popular normalization algorithms, although others have been developed [18,19].”
MethodDeep Sequencing the Transcriptome Reveals Seasonal Adaptive Mechanisms in a Hibernating Mammal
Hampton, M. E., Melvin, R. G., Kendall, A. H., Kirkpatrick, B. R., Peterson, N., & Andrews, M. T. (2011) · PLoS ONE · doi:10.1371/journal.pone.0027021 · Cited by 93
Comparison of transcriptomic landscapes of bovine embryos using RNA-Seq
Huang, W., & Khatib, H. (2010) · BMC Genomics · doi:10.1186/1471-2164-11-711 · Cited by 98
Genes, behavior and next‐generation RNA sequencing
Hitzemann, R., Bottomly, D., Darakjian, P., Walter, N., Iancu, O., Searles, R., Wilmot, B., & McWeeney, S. (2013) · Genes, Brain and Behavior · doi:10.1111/gbb.12007 · Cited by 88
Rutowicz, K., Lirski, M., Mermaz, B., Schubert, J., Teano, G., Mestiri, I., Kroteń, M. A., Fabrice, T. N., Fritz, S., Grob, S., et al. (2019) · Genome biology · doi:10.1186/s13059-019-1767-3 · Cited by 98
Lamin B2 Levels Regulate Polyploidization of Cardiomyocyte Nuclei and Myocardial Regeneration
Han, L., Choudhury, S., Mich-Basso, J. D., Ammanamanchi, N., Ganapathy, B., Suresh, S., Khaladkar, M., Singh, J. M., Maehr, R., Zuppo, D. A., et al. (2020) · Developmental Cell · doi:10.1016/j.devcel.2020.01.030 · Cited by 94
PIN6 is required for nectary auxin response and short stamen development
Bender, R. L., Fekete, M. L., Klinkenberg, P. M., Hampton, M. E., Bauer, B., Malecha, M. L., Lindgren, K., Maki, J. A., Perera, M. A. D. N., Nikolau, B. J., et al. (2013) · The Plant Journal · doi:10.1111/tpj.12184 · Cited by 94
Nicolle, R., Blum, Y., Duconseil, P., Vanbrugghe, C., Brandone, N., Poizat, F., Roques, J., Bigonnet, M., Gayet, O., Rubis, M., et al. (2020) · EBioMedicine · doi:10.1016/j.ebiom.2020.102858 · Cited by 93
“Gene expression pro fi les were obtained using FeatureCount and normalized using the upper-quartile approach 9 . tumour differentiation was de fi ned based on the following established criteria, brie fl y: tumors were considered poorly differentiated when tissue architecture is solid, forming massive…”
Cai, L., Liu, H., Huang, F., Fujimoto, J., Girard, L., Chen, J., Li, Y., Zhang, Y.-A., Deb, D., Stastny, V., et al. (2021) · Communications Biology · doi:10.1038/s42003-021-01842-7 · Cited by 93
White, M. A., Kitano, J., & Peichel, C. L. (2015) · Molecular Biology and Evolution · doi:10.1093/molbev/msv078 · Cited by 92
Hampton, M. E., Melvin, R. G., & Andrews, M. T. (2013) · PLoS ONE · doi:10.1371/journal.pone.0085157 · Cited by 91
Lopez-Maestre, H., Brinza, L., Marchet, C., Kielbassa, J., Bastien, S., Boutigny, M., Monnin, D., Filali, A. E., Carareto, C. M. A., Vieira, C. P., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw655 · Cited by 104
“The test was performed using a Likelihood Ratio Test with one degree of freedom, which corresponds to the supplementary interaction parameter that is included in the second model and not in the first (25).”
MethodLi, L., Li, D., Liu, S., Ma, X., Dietrich, C. R., Hu, H.-C., Zhang, G., Liu, Z., Zheng, J., Wang, G., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0082333 · Cited by 91
Liu, R., Loraine, A. E., & Dickerson, J. A. (2014) · BMC Bioinformatics · doi:10.1186/s12859-014-0364-4 · Cited by 95
Pyicos: a versatile toolkit for the analysis of high-throughput sequencing data
Althammer, S., González‐Vallinas, J., Ballaré, C., Beato, M., & Eyras, E. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr570 · Cited by 90
Zhao, J., Sun, Y., Huang, Y., Song, F., Huang, Z., Bao, Y., Zuo, J., Saffen, D. W., Shao, Z., Liu, W., et al. (2017) · Scientific Reports · doi:10.1038/srep40488 · Cited by 90
The pause-initiation limit restricts transcription activation in human cells
Gressel, S., Schwalb, B., & Cramer, P. (2019) · Nature Communications · doi:10.1038/s41467-019-11536-8 · Cited by 94
McCormick, K. P., Willmann, M. R., & Meyers, B. C. (2011) · Silence · doi:10.1186/1758-907x-2-2 · Cited by 92
A Local Poisson Graphical Model for Inferring Networks From Sequencing Data
Allen, G. I., & Liu, Z. (2013) · IEEE Transactions on NanoBioscience · doi:10.1109/tnb.2013.2263838 · Cited by 88
Wang, N., Wang, R., Wang, R., & Chen, S. (2018) · Functional & Integrative Genomics · doi:10.1007/s10142-018-0595-y · Cited by 88
“As numerous studies have underscored the importance of biological and technical replicates within RNA-seq (Bullard et al. 2010; Auer and Doerge 2010; McIntyre et al. 2011), we selected 15 biological and three technical replicates for each tissue to achieve more reliable results.”
Effects of library size variance, sparsity, and compositionality on the analysis of microbiome dataPreprint
Weiss, S. J., Xu, Z. Z., Amir, A., Peddada, S., Bittinger, K. L., Gonzalez, A., Lozupone, C., Zaneveld, J., Vázquez‐Baeza, Y., Birmingham, A., et al. (2015) · doi:10.7287/peerj.preprints.1157v1 · Cited by 88
ROTS: reproducible RNA-seq biomarker detector—prognostic markers for clear cell renal cell cancer
Seyednasrollah, F., Rantanen, K., Jaakkola, P., & Elo, L. L. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkv806 · Cited by 75
“The necessity of normalization has been proved in RNAseq studies (20,21).”
BackgroundTranscriptome profiling of human hippocampus dentate gyrus granule cells in mental illness
Kohen, R., Dobra, A., Tracy, J. H., & Haugen, E. (2014) · Translational Psychiatry · doi:10.1038/tp.2014.9 · Cited by 87
panelcn.MOPS: Copy-number detection in targeted NGS panel data for clinical diagnostics
Povysil, G., Tzika, A., Vogt, J., Haunschmid, V., Messiaen, L. M., Zschocke, J., Klambauer, G., Hochreiter, S., & Wimmer, K. (2017) · Human Mutation · doi:10.1002/humu.23237 · Cited by 88
“The best performance was achieved by normalizing to the third quartile, which is equivalent to the upper quartilenormalization(Bullard,Purdom,Hansen, &Dudoit,2010),and rescaling with the first quartile of the scaling factors.”
Removing unwanted variation from large-scale RNA sequencing data with PRPS
Molania, R., Foroutan, M., Gagnon-Bartsch, J. A., Gandolfo, L. C., Jain, A., Sinha, A., Olshansky, G., Dobrovic, A., Papenfuss, A. T., & Speed, T. P. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01440-w · Cited by 92
Li, X., Brock, G. N., Rouchka, E. C., Cooper, N. G. F., Wu, D., O’Toole, T. E., Gill, R. S., Eteleeb, A. M., O’Brien, L., & Rai, S. N. (2017) · PLoS ONE · doi:10.1371/journal.pone.0176185 · Cited by 89
“One reason is that a small number of highly expressed genes can consume a significant amount of the total sequence [18].”
BackgroundMethodInfluentialRecombination affects accumulation of damaging and disease-associated mutations in human populations
Hussin, J., Hodgkinson, A., Idaghdour, Y., Grenier, J., Goulet, J.-P., Gbeha, E., Hip-Ki, E., & Awadalla, P. (2015) · Nature Genetics · doi:10.1038/ng.3216 · Cited by 85
Zhang, P., Li, C., Zhu, L., Su, X., Li, Y., Jin, C., & Li, T. (2013) · PLoS ONE · doi:10.1371/journal.pone.0073506 · Cited by 85
Su, Z., Li, Z., Chen, T., Li, Q., Fang, H., Ding, D., Ge, W., Ning, B., Hong, H., Perkins, R., et al. (2011) · Chemical Research in Toxicology · doi:10.1021/tx200103b · Cited by 86
Technical Variations in Low-Input RNA-seq Methodologies
Bhargava, V., Head, S. R., Ordoukhanian, P. T., Mercola, M. A., & Subramaniam, S. (2014) · Scientific Reports · doi:10.1038/srep03678 · Cited by 86
Evaluation of methods for differential expression analysis on multi-group RNA-seq count data
Tang, M., Sun, J., Shimizu, K., & Kadota, K. (2015) · BMC Bioinformatics · doi:10.1186/s12859-015-0794-7 · Cited by 83
“Read counts across technical replicates derived from a single source fit to a Poisson distribution [3, 21].”
BackgroundGKAP Acts as a Genetic Modulator of NMDAR Signaling to Govern Invasive Tumor Growth
Li, L., Zeng, Q., Bhutkar, A. J., Galván, J. A., Karamitopoulou, E., Noordermeer, D., Peng, M.-W., Piersigilli, A., Perren, A. A., Zlobec, I., et al. (2018) · Cancer Cell · doi:10.1016/j.ccell.2018.02.011 · Cited by 84
“Raw estimated expression counts were upper-quartile normalized to a count of 1000 (Bullard et al., 2010).”
Leng, N., Li, Y., McIntosh, B. E., Nguyen, B. K., Duffin, B. M., Tian, S., Thomson, J. A., Dewey, C. N., Stewart, R. M., & Kendziorski, C. M. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv193 · Cited by 85
Góngora-Castillo, E., Ibarra-Laclette, E., Trejo-Saavedra, D., & Rivera-Bustamante, R. (2012) · Virology Journal · doi:10.1186/1743-422x-9-295 · Cited by 75
Deep sequencing-based analysis of the anaerobic stimulon in Neisseria gonorrhoeae
Isabella, V. M., & Clark, V. L. (2011) · BMC Genomics · doi:10.1186/1471-2164-12-51 · Cited by 84
PipX, the coactivator of NtcA, is a global regulator in cyanobacteria
Espinosa, J., Rodríguez, F., Salinas, P., Lanza, V. F., Dixon, R. A., de la Cruz, F., & Contreras, A. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1404097111 · Cited by 84
Robustness of differential gene expression analysis of RNA-seq
Stupnikov, A., McInerney, C. E., Savage, K. I., McIntosh, S., Emmert‐Streib, F., Kennedy, R. D., Salto‐Tellez, M., Prise, K. M., & McArt, D. G. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.05.040 · Cited by 84
“DGE method’s performance has also been compared by validating the expression of true positive results using real-time quantitative PCR (RT-qPCR) [26,32].”
BackgroundMethodConcepcion, C. P., Ma, S., LaFave, L. M., Bhutkar, A. J., Liu, M., DeAngelo, L. P., Kim, J. Y., Priore, I. S. D., Schoenfeld, A. J., Miller, M., et al. (2022) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 84
IL-33 Signaling Alters Regulatory T Cell Diversity in Support of Tumor Development
Li, A., Herbst, R. H., Canner, D. A., Schenkel, J. M., Smith, O. C. G., Kim, J. Y., Hillman, M., Bhutkar, A. J., Cuoco, M. S., Rappazzo, C. G., et al. (2019) · Cell Reports · doi:10.1016/j.celrep.2019.10.120 · Cited by 83
“counts estimated from RSEM were upper-quartile normalized to a count of 1000 per sample (Bullard et al., 2010).”
MethodFrom Smart Health to Smart Hospitals
Holzinger, A., Röcker, C., & Ziefle, M. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16226-3_1 · Cited by 84
Papini, M., Nookaew, I., Uhlén, M., & Nielsen, J. (2012) · Microbial Cell Factories · doi:10.1186/1475-2859-11-136 · Cited by 83
MYC activation cooperates with Vhl and Ink4a/Arf loss to induce clear cell renal cell carcinoma
Bailey, S. T., Smith, A. M., Kardos, J., Wobker, S. E., Wilson, H. L., Krishnan, B., Saito, R., Lee, H. J., Zhang, J., Eaton, S. C., et al. (2017) · Nature Communications · doi:10.1038/ncomms15770 · Cited by 86
Sinicropi, D. V., Qu, K., Collin, F., Crager, M., Liu, M., Pelham, R. J., Pho, M., Rossi, A. D., Jeong, J., Scott, A., et al. (2012) · PLoS ONE · doi:10.1371/journal.pone.0040092 · Cited by 82
Long noncoding RNA repertoire in chicken liver and adipose tissue
Muret, K., Klopp, C., Wucher, V., Esquerré, D., Legeai, F., Lecerf, F., Désert, C., Boutin, M., Jehl, F., Acloque, H., et al. (2017) · Genetics Selection Evolution · doi:10.1186/s12711-016-0275-0 · Cited by 82
“The raw counts for each gene were calculated by feature-Counts [72] at the gene (locus) level and normalized by the gene size and the total number of reads that mapped in the most highly expressed genes, as proposed in the upper quartile (UQ) method described by Bullard et al. [77].”
MethodHanna, A., Metge, B. J., Bailey, S. K., Chen, D., Chandrashekar, D. S., Varambally, S., Samant, R. S., & Shevde, L. A. (2018) · OncoImmunology · doi:10.1080/2162402x.2018.1548241 · Cited by 85
Elbashir, M. K., Ezz, M. M., Mohammed, M., & Saloum, S. S. (2019) · IEEE Access · doi:10.1109/access.2019.2960722 · Cited by 83
Ontogeny of the Maize Shoot Apical Meristem
Takacs, E. M., Li, J., Du, C., Ponnala, L., Janick‐Buckner, D., Yu, J., Muehlbauer, G. J., Schnable, P. S., Timmermans, M. C. P., Sun, Q., et al. (2012) · The Plant Cell · doi:10.1105/tpc.112.099614 · Cited by 82
Tausta, S. L., Li, P., Si, Y., Gandotra, N., Liu, P., Sun, Q., Brutnell, T. P., & Nelson, T. M. (2014) · Journal of Experimental Botany · doi:10.1093/jxb/eru152 · Cited by 78
The transcriptional landscape of Chlamydia pneumoniae
Albrecht, M., Sharma, C. M., Dittrich, M. T., Müller, T., Reinhardt, R., Vogel, J., & Rudel, T. (2011) · Genome biology · doi:10.1186/gb-2011-12-10-r98 · Cited by 81
Roy, S., Hooiveld, G., Seehawer, M., Caruso, S., Heinzmann, F., Schneider, A., Frank, A., Cardenas, D. V., Sonntag, R., Luedde, M., et al. (2018) · Gastroenterology · doi:10.1053/j.gastro.2018.08.032 · Cited by 82
Fernández-Martínez, A., Pascual, T., Singh, B., Nuciforo, P. G., Rashid, N. U., Ballman, K. V., Campbell, J. D., Hoadley, K. A., Spears, P. A., Paré, L., et al. (2023) · JAMA Oncology · doi:10.1001/jamaoncol.2022.6288 · Cited by 80
Metadata
- Authors
- Bullard, J. H., Purdom, E., Hansen, K. D., & Dudoit, S.
- Affiliations
- University of California, Berkeley (US)
- Corresponding author
- Bullard, James H
- Venue
- BMC Bioinformatics
- Published
- 2010-02-18
- Volume
- 11
- Issue
- 1
- Pages
- 94-94
- ISSN
- 1471-2105 (ISSN-L)
- PMC
- PMC2838869
- OpenAlex
- W2156631105
- Cited by
- 1,680
- References
- 23
- Citations by year
- 2026: 29 · 2025: 51 · 2024: 69 · 2023: 68 · 2022: 92 · 2021: 132 · 2020: 127 · 2019: 123 · 2018: 123 · 2017: 133 · 2016: 121 · 2015: 129 · 2014: 161 · 2013: 128 · 2012: 98
- FWCI
- 41.79
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Bioinformatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Collection of Biostatistics Research Archive (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- National Science Foundation
- National Institutes of Health
- National Human Genome Research Institute (U01HG004271)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Keywords
- mRNA-seq, read count normalization, differential gene expression, method comparison
- MeSH terms
- RNA, Messenger, Sequence Analysis, RNA, Computational Biology, Databases, Genetic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
