Assessing technical performance in differential gene expression experiments with external spike-in RNA control ratio mixtures
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
There is a critical need for standard approaches to assess, report and compare the technical performance of genome-scale differential gene expression experiments. Here we assess technical performance with a proposed standard ‘dashboard’ of metrics derived from analysis of external spike-in RNA control ratio mixtures. These control ratio mixtures with defined abundance ratios enable assessment of diagnostic performance of differentially expressed transcript lists, limit of detection of ratio (LODR) estimates and expression ratio variability and measurement bias. The performance metrics suite is applicable to analysis of a typical experiment, and here we also apply these metrics to evaluate technical performance among laboratories. An interlaboratory study using identical samples shared among 12 laboratories with three different measurement processes demonstrates generally consistent diagnostic power across 11 laboratories. Ratio measurement variability and bias are also comparable among laboratories for the same measurement process. We observe different biases for measurement processes using different mRNA-enrichment protocols. Differential gene expression experiments yield quantitative insight into biological activity and may be important in disease classification and treatment. Here, the authors analyse external spike-in RNA controls to provide a standard method to assess and compare experiment performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
McCarthy, D. J., Chen, Y., & Smyth, G. K. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks042 · Cited by 5,195
Self-citationHuber, W., von Heydebreck, A., Sültmann, H., Poustka, A., & Vingron, M. (2002) · Bioinformatics · doi:10.1093/bioinformatics/18.suppl_1.s96 · Cited by 2,440
MAQC Consortium, Shi, L., Shi, L., Reid, L. H., Jones, W. D., Shippy, R., Warrington, J. A., Baker, S. C., Collins, P. J., de Longueville, F., et al. (2006) · Nature Biotechnology · doi:10.1038/nbt1239 · Cited by 2,126
Self-citationSEQC/MAQC-III Consortium (2014) · Nature Biotechnology · doi:10.1038/nbt.2957 · Cited by 983
Self-citationRNA-SeQC: RNA-seq metrics for quality control and process optimization
DeLuca, D. S., Levin, J. Z., Sivachenko, A., Fennell, T., Nazaire, M.-D., Williams, C., Reich, M., Winckler, W., & Getz, G. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts196 · Cited by 899
Synthetic spike-in standards for RNA-seq experiments
Jiang, L., Schlesinger, F., Davis, C. A., Zhang, Y., Li, R., Salit, M., Gingeras, T. R., & Oliver, B. (2011) · Genome Research · doi:10.1101/gr.121095.111 · Cited by 653
Self-citationRevisiting Global Gene Expression Analysis
Lovén, J., Orlando, D. A., Sigova, A. A., Lin, C. Y., Rahl, P. B., Burge, C. B., Levens, D. L., Lee, T. I., & Young, R. A. (2012) · Cell · doi:10.1016/j.cell.2012.10.012 · Cited by 616
Wang, C., Gong, B., Bushel, P. R., Thierry-Mieg, J., Thierry-Mieg, D., Xu, J., Fang, H., Hong, H., Shen, J., Su, Z., et al. (2014) · Nature Biotechnology · doi:10.1038/nbt.3001 · Cited by 512
Self-citationThe External RNA Controls Consortium: a progress report
The External RNA Controls Consortium (2005) · Nature Methods · doi:10.1038/nmeth1005-731 · Cited by 373
Li, S., Tighe, S. W., Nicolet, C. M., Grove, D., Levy, S., Farmerie, W., Viale, A., Wright, C., Schweitzer, P. A., Gao, Y., et al. (2014) · Nature Biotechnology · doi:10.1038/nbt.2972 · Cited by 240
Self-citationDetecting and correcting systematic variation in large-scale RNA sequencing data
Li, S., Łabaj, P. P., Zumbo, P., Sykacek, P., Shi, W., Shi, L., Phan, J., Wu, P.-Y., Wang, M., Wang, C., et al. (2014) · Nature Biotechnology · doi:10.1038/nbt.3000 · Cited by 175
Self-citationUsing RNA sample titrations to assess microarray platform performance and normalization techniques
Shippy, R., Fulmer-Smentek, S., Jensen, R. V., Jones, W. D., Wolber, P. K., Johnson, C. D., Pine, P. S., Boysen, C., Guo, X., Chudin, E., et al. (2006) · Nature Biotechnology · doi:10.1038/nbt1241 · Cited by 173
Self-citationCaveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them)
Berrar, D., & Flach, P. (2012) · Briefings in Bioinformatics · doi:10.1093/bib/bbr008 · Cited by 136 · cited 4×
“Examples of the erccdashboard performance measure figures are presented in Figs 2–5 for two arbitrarily-selected example experiments from the large SEQC and ABRF experiment cohorts (for all results see Supplementary Figs 1–20).”
BackgroundMethodInfluential
Research works printed but not linked: 23
- Salit, M. Standards in gene expression microarray experiments. Methods Enzymol. 411, 63–78 (2006). · DOI
- Lippa, K. A., Duewer, D. L., Salit, M. L., Game, L. & Causton, H. C. Exploring the use of internal and external controls for assessing microarray technical performance. BMC Res. Notes 3, 349 (2010). · DOI
- McCall, M. N. & Irizarry, R. A. Consolidated strategy for the analysis of microarray spike-in data. Nucleic Acids Res. 36, e108 (2008). · DOI
- Tong, W. et al. Evaluation of external RNA controls for the assessment of microarray performance. Nat. Biotechnol. 24, 1132–1139 (2006). · DOI
- van de Peppel, J. et al. Monitoring global messenger RNA changes in externally controlled microarray experiments. EMBO Rep. 4, 387–393 (2003). · DOI · Paper page
- Adiconis, X. et al. Comparative analysis of RNA sequencing methods for degraded or low-input samples. Nat. methods 10, 623–629 (2013). · DOI · Paper page
- Gertz, J. et al. Transposase mediated construction of RNA-seq libraries. Genome Res. 22, 134–141 (2012). · DOI
- Griffith, M. et al. Alternative expression analysis by RNA sequencing. Nat. Methods 7, 843–847 (2010). · DOI · Paper page
- Ramskold, D. et al. Full-length mRNA-Seq from single-cell levels of RNA and individual circulating tumor cells. Nat. Biotechnol. 30, 777–782 (2012). · DOI · Paper page
- Novoradovskaya, N. et al. Universal Reference RNA as a standard for microarray experiments. BMC Genomics 5, 20 (2004). · DOI · Paper page
- Pine, P. S. et al. Use of diagnostic accuracy as a metric for evaluating laboratory proficiency with microarray assays using mixed-tissue RNA reference samples. Pharmacogenomics 9, 1753–1763 (2008). · DOI
- Dudoit, S., Yang, Y. H., Callow, M. J. & Speed, T. P. Statistical methods for identifying differentially expressed genes in replicated cDNA microarray experiments. Stat. Sin. 12, 111–139 (2002).
- Liao, Y., Smyth, G. K. & Shi, W. The Subread aligner: fast, accurate and scalable read mapping by seed-and-vote. Nucleic Acids Res. 41, e108 (2013). · DOI · Paper page
- Liao, Y., Smyth, G. K. & Shi, W. featureCounts: an efficient general purpose program for assigning sequence reads to genomic features. Bioinformatics 30, 923–930 (2014). · DOI · Paper page
- Trapnell, C., Pachter, L. & Salzberg, S. L. TopHat: discovering splice junctions with RNA-Seq. Bioinformatics 25, 1105–1111 (2009). · DOI · Paper page
- Li, H. & Durbin, R. Fast and accurate short read alignment with Burrows-Wheeler transform. Bioinformatics 25, 1754–1760 (2009). · DOI · Paper page
- Fasold, M., Stadler, P. F. & Binder, H. G-stack modulated probe intensities on expression arrays—sequence corrections and signal calibration. BMC Bioinformatics 11, (2010). · DOI
- Hochreiter, S., Clevert, D. A. & Obermayer, K. A new summarization method for affymetrix probe level data. Bioinformatics 22, 943–949 (2006). · DOI · Paper page
- Mueckstein, U., Leparc, G. G., Posekany, A., Hofacker, I. & Kreil, D. P. Hybridization thermodynamics of NimbleGen microarrays. BMC Bioinformatics 11, 35 (2010). · DOI
- Gentleman, R. C. et al. Bioconductor: open software development for computational biology and bioinformatics. Genome Biol. 5, R80 (2004). · DOI · Paper page
- Lund, S., Nettleton, D., McCarthy, D. & Smyth, G. Detecting differential expression in RNA-sequence data using quasi-likelihood with shrunken dispersion estimates. Stat. Appl. Genet. Mol. Biol. 11,, Iss. 5, Art. 8 (2012). · DOI · Paper page
- Robinson, M. D., McCarthy, D. J. & Smyth, G. K. edgeR: a Bioconductor package for differential expression analysis of digital gene expression data. Bioinformatics 26, 139–140 (2010). · DOI · Paper page
- Wickham, H. ggplot2: Elegant Graphics for Data Analysis Springer (2009). · DOI · Paper page
Cited by
Clinical Metagenomic Next-Generation Sequencing for Pathogen Detection
Gu, W., Miller, S., & Chiu, C. Y. (2019) · Annual Review of Pathology: Mechanisms of Disease · doi:10.1146/annurev-pathmechdis-012418-012751 · Cited by 1,458
SEQC/MAQC-III Consortium (2014) · Nature Biotechnology · doi:10.1038/nbt.2957 · Cited by 983
Self-citationTranslating RNA sequencing into clinical diagnostics: opportunities and challenges
Byron, S. A., Van Keuren-Jensen, K. R., Engelthaler, D. M., Carpten, J. D., & Craig, D. W. (2016) · Nature Reviews Genetics · doi:10.1038/nrg.2016.10 · Cited by 809
variancePartition: interpreting drivers of variation in complex gene expression studies
Hoffman, G. E., & Schadt, E. E. (2016) · BMC Bioinformatics · doi:10.1186/s12859-016-1323-z · Cited by 769
Revealing the vectors of cellular identity with single-cell genomics
Wagner, A., Regev, A., & Yosef, N. (2016) · Nature Biotechnology · doi:10.1038/nbt.3711 · Cited by 632
Metadata
- Authors
- Munro, S. A., Lund, S. P., Pine, P. S., Binder, H., Clevert, D.-A., Conesa, A., Dopazo, J., Fasold, M., Hochreiter, S., Hong, H., Jafari, N., Kreil, D. P., Łabaj, P. P., Li, S., Liao, Y., Lin, S. M., Meehan, J., Mason, C. E., Santoyo-Lopez, J., Setterquist, R. A., Shi, L., Shi, W., Smyth, G. K., Stralis-Pavese, N., Su, Z., Tong, W., Wang, C., Wang, J., Xu, J., Ye, Z., Yang, Y., Yu, Y., & Salit, M.
- Venue
- Nature Communications
- Published
- 2014-09-25
- Volume
- 5
- Issue
- 1
- Pages
- 5125-5125
- ISSN
- 2041-1723 (ISSN-L)
- OpenAlex
- W2151876471
- Cited by
- 124
- References
- 13
- FWCI
- 4.87
- Type
- journal article
- Language
- en
- Open access
- bronze
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.