CoChat
Sign Up
Cited by 175Gene expression and cancer classificationGenomics and Phylogenetic StudiesSingle-cell and spatial transcriptomics

Detecting and correcting systematic variation in large-scale RNA sequencing data

Open access
Li, Sheng, Łabaj, Paweł P, Zumbo, Paul, Sykacek, Peter, Shi, Wei, Shi, Leming, Phan, John, Wu, Po-Yen +6 more·2014·Nature Biotechnology
DOI 10.1038/nbt.3000PMC PMC4160374
Open this paper in CoChat — free account

Abstract

High-throughput RNA sequencing (RNA-seq) enables comprehensive scans of entire transcriptomes, but best practices for analyzing RNA-seq data have not been fully defined, particularly for data collected with multiple sequencing platforms or at multiple sites. Here we used standardized RNA samples with built-in controls to examine sources of error in large-scale RNA-seq studies and their impact on the detection of differentially expressed genes (DEGs). Analysis of variations in guanine-cytosine content, gene coverage, sequencing error rate and insert size allowed identification of decreased reproducibility across sites. Moreover, commonly used methods for normalization (cqn, EDASeq, RUV2, sva, PEER) varied in their ability to remove these systematic biases, depending on sample complexity and initial data quality. Normalization methods that combine data from genes across sites are strongly recommended to identify and remove site-specific effects and can substantially improve RNA-seq studies.

Integrity signals

Registry

Crossref · 10.1038/nbt.3000

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 30 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Inter-site comparisons of identical RNA samples on Illumina HiSeq2000 produced many false positive DEGs at lenient thresholds (FC 1.5, FDR 0.05), up to 9,602 (mean 2,823, SD 3,527), about 20% of genes.

    “At the most lenient FC (1.5) and FDR (0.05), the number of false positive DEGs detected was as high as 9,602 (mean = 2,823, S.D. = 3,527, including both changes up and down), or ∼20% of all genes (Fig. 1b).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Identical RNA samples processed at different sites should produce little and random expression variation.
  • Findingown data

    In large-scale RNA-seq studies, commonly used normalization methods (cqn, EDASeq, RUV2, sva, PEER) differ in how well they remove systematic biases, depending on sample complexity and initial data quality.

    “commonly used methods fornormalization (cqn, EDASeq, RUV2, sva, PEER) varied in their ability to remove these systematic biases, depending on sample complexity and initial data quality.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [14] Hansen, K.D., Irizarry, R.A. & Wu, Z. Removing technical variability in RNA-seq data using conditional quantile normalization. Biostatistics 13, 204–216 (2012). source

      Not checked in the free run.

    • cites [15] Risso, D., Schwartz, K., Sherlock, G. & Dudoit, S. GC-content normalization for RNA-Seq data. BMC Bioinformatics 12, 480 (2011). source

      Not checked in the free run.

    • cites [31] Gagnon-Bartsch, J.A. & Speed, T.P. Using control genes to correct for unwanted variation in microarray data. Biostatistics 13, 539–552 (2012). source

      Not checked in the free run.

  • Findingown data

    Even at stringent thresholds (FC >2.0, FDR <0.001), inter-site comparisons of identical samples yielded up to 3,135 false positive DEGs (mean 739, SD 1,089), up to 8% of genes.

    “although even at fairly stringent thresholds (FC >2.0 and FDR <0.001), the number of DEGs detected was still as high as 3,135 (mean = 739, S.D. = 1,089), representing up to 8% of all genes.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Identical RNA samples processed at different sites should produce little and random expression variation.
  • Findingown data

    High inter-site false positive DEG rates were observed across multiple analysis pipelines (WHAM, Mapsplice, Novoalign, Cufflinks, HTSeq), independent of pipeline or alignment method.

    “When we examined the inter-site DEG false positive rates for several other analysis pipelines (WHAM34, Mapsplice35, Novoalign36, Cufflinks23, 37, 38 and HTSeq39) we found similarly high false positive rates, regardless of the analysis pipeline or read alignment methods used (Supplementary Fig. 1).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [34] Li, Y., Terrell, A. & Patel, J.M. in Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data 445–456 (ACM, 2011). source

      Not checked in the free run.

    • cites [35] Wang, K. et al. MapSplice: accurate mapping of RNA-seq reads for splice junction discovery. Nucleic Acids Res. 38, e178 (2010). source

      Not checked in the free run.

    • cites [36] Li, H. & Homer, N. A survey of sequence alignment algorithms for next-generation sequencing. Brief. Bioinform. 11, 473–483 (2010). source

      Not checked in the free run.

    • Rests on an assumption: Identical RNA samples processed at different sites should produce little and random expression variation.
  • Findingown data

    PEER sometimes ranked lower on comparability to the TaqMan reference set.

    “although PEER sometimes ranked lower on comparability to the Taqman reference set”
    Quote in the paperNot a substantial claim
  • May combine two claims. 24
  • Classified as probably not a substantial claim. 6
  • The quote only partly states this claim. 2

  • May combine two claims.

    Findingown data

    Site ILM3 consistently showed the lowest Spearman correlation of p-values and an increase in site-specific DEGs.

    “As expected, one site (ILM3) always showed the lowest Spearman correlation of p-values (Fig/ 2a), co-incident with an increase in site-specific DEGs (Fig. 2).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All six Illumina sites found similar numbers of DEGs, and for DEGs common to all six sites, inter-site rank agreement of p-values was very high (median Spearman correlation >0.96).

    “All six sites found similar numbers of DEGs (Supplementary Fig. 9), and the Spearman rank correlation of p-values showed that the inter-site rank agreement was very high for the common DEGs shared by all 6 sites, with a median correlation greater than 0.96 (Supplementary Fig. 10a–c).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    PEER consistently performed better than other methods when common intra-site DEGs were used to validate inter-site DEGs, especially at ILM3, where it identified and compensated the bias for every comparison.

    “Using the common intra-site DEGs to validate inter-site DEGs showed that PEER consistently performed better, especially for the site with the largest bias (ILM3), where PEER successfully identified and compensated this bias, for every comparison (Supplemental Figure 15).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    DEG agreement between RNA-seq and TaqMan decreased as samples became more similar, with the largest differences expected for A vs D and the smallest for C vs D.

    “Each of the six cross-sample comparisons had very similar MCC, TPR and FPR (Fig. 2 and Supplementary Fig. 13), and these measures also indicated lower agreement as the samples became more similar, as expected, with the biggest differences expected by design in comparisons of samples A and D, whereas conversely the mixture samples C and D were similar by design.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: TaqMan measurements provide a reliable reference standard for true DEG status.

  • May combine two claims.

    Findingown data

    Site ILM3 had higher coefficients of variation of coverage across gene bodies, demonstrating the value of a nucleotide composition metric for QC.

    “Coverage across the gene body was assessed using the coefficient of variation of the coverage across the length of the genes, and we saw that overall the ILM3 site had higher coefficients of variation (Fig. 5d and Supplementary Fig. 18c), thus demonstrating the value of a ‘nucleotide composition metric’ (described below) for QC in RNA-seq for identifying unusual and potentially problematic measur”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Latent experimental factors identified by PEER and sva are highly correlated with QC metrics and were responsible for the majority of false positives in inter-site DEG analysis.

    “Finally, we observed that the latent experimental factors determined by PEER and sva are highly correlated with QC metrics and properties, and that these factors were responsible for the majority of false positives in inter-site DEG analysis.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    After TMM and limma-voom normalization, samples A and B were well separated by multidimensional scaling, and two replicates with abnormal GC distributions were separated from other sample A replicates at dimension 2.

    “After the Trimmmed Mean of M-values (TMM)55 and limma-voom normalization, we found that samples A and B were well distinguished by multidimensional scaling (Supplementary Fig. 21d), and that the two replicates with abnormal GC content distributions (PGM1.A.4 and PGM2.A.2) were separated from the other replicates of sample A at dimension 2.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [55] Li, H. et al. The Sequence Alignment/Map format and SAMtools. Bioinformatics 25, 2078–2079 (2009). source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Applying GC bias correction tools (cqn, EDASeq) to individual sites or restricting read counts to 3′-UTRs did not reduce inter-site false positive DEGs and often increased them.

    “Notably, neither applying GC bias correction tools (cqn14 and EDASeq15) to individual sites nor changing read counts to only use 3′-UTRs was effective at decreasing the number of inter-site false positive DEGs (Fig. 1b and Supplementary Fig. 3); in most cases, these methods actually increased the number of false positives.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [14] Hansen, K.D., Irizarry, R.A. & Wu, Z. Removing technical variability in RNA-seq data using conditional quantile normalization. Biostatistics 13, 204–216 (2012). source

      Not checked in the free run.

    • cites [15] Risso, D., Schwartz, K., Sherlock, G. & Dudoit, S. GC-content normalization for RNA-Seq data. BMC Bioinformatics 12, 480 (2011). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    All normalization methods showed reduced inter-site reproducibility as samples became more similar.

    “All methods yielded reduced inter-site reproducibility as samples become more similar, as expected.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    RUV2 with ERCC spike-ins removed on average about 20% of false positives and was more effective at sites with already low false positive rates (ILM4, ILM5).

    “The application of RUV2 with ERCC spike-ins (RUV2-ERCC), which tries to remove confounding factors based on a control set of synthetic RNAs assessed across sites, on average removes just 20% of false positives on average, but is more effective for sites that already have relatively low false positive rates (ILM4, ILM5).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [31] Gagnon-Bartsch, J.A. & Speed, T.P. Using control genes to correct for unwanted variation in microarray data. Biostatistics 13, 539–552 (2012). source

      Not checked in the free run.

  • May combine two claims.

    Findingreasoning

    Reproducibility of intra- and inter-site DEGs was higher for more biologically different samples (A vs B) and lower for more similar samples (A vs C, B vs D, C vs D).

    “Overall, the reproducibility of intra- and inter-site DEGs across all sites showed a higher correlation for comparisons between more biologically different samples (A versus B), and a lower reproducibility for more similar samples (A versus C, B versus D, C versus D), reflecting the expected greater challenge of reliably identifying smaller differences.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    EDASeq gave the highest similarity to TaqMan expression measures for intra-site DEG detection (mean MCC = 0.939, SD = 0.019), while most methods did not improve intra-site DEG accuracy.

    “Although most methods did not improve the accuracy of intra-site DEGs detection, we found that EDAseq gave the highest similarity to Taqman expression measures (with mean MCC = 0.939 and S.D. = 0.019, Fig. 4 and Supplementary Figure 16b).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: TaqMan measurements provide a reliable reference standard for true DEG status.

  • The quote only partly states this claim.

    Findingown data

    RUV factor analysis based on the ERCC control gene set was not sufficient to remove the false positives.

    “Similarly, RUV factor analysis based on the ERCC control gene set was not sufficient.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A non-random nucleotide composition bias at the beginning of sequencing reads and a distinct narrow GC-content bell curve were observed at site ILM3.

    “First, a non-random nucleotide composition bias was seen at the beginning of the sequencing reads, concomitant with a distinct, narrow bell curve of GC-content for the IML3 site (Fig. 5a).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Intra-site and inter-site Pearson correlations of gene expression were all above 0.95, and expression distributions from different sites clustered together.

    “First, the intra-site and inter-site Pearson correlation coefficients (R2) were all above 0.95 (Supplementary Fig. 4), and Q-Q plots of the gene expression values from different sites indicated that all sites had similar distributions that clustered together (Supplementary Figs. 5–8).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For sva, the first latent factor was significantly correlated with GC content distribution, average error rate, and duplication by library; the second was associated with gene-body coverage uniformity.

    “For sva, the first latent factor was significantly correlated with the GC content distribution quality metric of the sites (p < 2×10-7), the average error rate (p < 6×10-7) and the duplication by library (see Supplementary Fig. 19p < 2×10-4). The second latent factor was significantly associated with the genebody coverage uniformity (p < 3×10-4).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The optimal number of hidden confounders was two for the ILM data set and three for the PGM data set.

    “The optimal number of hidden confounders was found to be two and three for ILM and PGM data sets respectively”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Normalization methods that pool data across genes and sites can identify and remove site-specific effects and can substantially improve RNA-seq studies.

    “Normalization methods that combine data from genes across sites are strongly recommended to identify and remove site-specific effects, and can substantially improve RNA-seq studies.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The cross-site internal control library showed that GC-content is preparation-specific rather than laboratory-specific, and the coefficient of variation for gene-body coverage is an important quality measure that quantifies 5′-3′ bias across both platforms.

    “Furthermore, our use of the cross-site, internal control library (#5) has demonstrated that GC-content is preparation-specific, not laboratory specific, and we have introduced the coefficient of variation for genebody coverage as an important quality measure in RNA-seq (Table 1), which quantifies this 5′-3′ bias across both platforms.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Study design and conductreasoning

    The fifth-library design allowed separation of variation from library preparation versus sequencing.

    “With this experimental design we can separate out sources of variation as either arising from the library preparation (including RNA isolation) or as arising from the sequencing itself.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    More than 85% of RNA-seq false positives could be removed by subtracting latent variable effects identified by sva or PEER through joint analysis of all genes across multiple sites, without decreasing sensitivity or specificity of DEG detection.

    “However, the majority of RNA-seq false positives (>85%) could successfully be removed by subtracting the effects of latent variables identified by either sva26,27 or PEER30, which could be achieved by jointly analyzing the set of measurements of all genes across multiple sites, without a decrease in the sensitivity or specificity of DEG detection at each site or across sites.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [26] Leek, J.T., Johnson, W.E., Parker, H.S., Jaffe, A.E. & Storey, J.D. The sva package for removing batch effects and other unwanted variation in high-throughput experiments. Bioinformatics 28, 882–883 (2012). source

      Not checked in the free run.

    • cites [27] Leek, J.T. & Storey, J.D. Capturing heterogeneity in gene expression studies by surrogate variable analysis. PLoS Genet. 3, 1724–1735 (2007). source

      Not checked in the free run.

    • cites [30] Stegle, O., Parts, L., Durbin, R. & Winn, J. A Bayesian framework to account for complex non-genetic factors in gene expression levels greatly increases power in eQTL studies. PLoS Comput. Biol. 6, e1000770 (2010). source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Sample B had more read coverage near the 3′ end than the 5′ end regardless of preparation or sequencing site, suggesting its stock may have been contaminated before poly(A)+ selection.

    “Plots of the uniformity of coverage across gene bodies showed that sample B, regardless of where it was prepared or sequenced, had more read coverage near the 3′ end than the 5′ end with respect to annotated gene models (Supplementary Fig. 18c), indicating that the stock of sample B prior to its distribution to each site may have been contaminated with something which would have caused it to have”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The findings confirm GC-content and insert size as recognized RNA-seq latent variables and identify gene coverage variation and error rate as additional contributions to technical variation.

    “This confirms the impact of two already recognized RNA-seq latent variables, GC-content and insert size15,22, and it also identifies two more relevant contributions to technical variation, gene coverage variation and error rate.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [15] Risso, D., Schwartz, K., Sherlock, G. & Dudoit, S. GC-content normalization for RNA-Seq data. BMC Bioinformatics 12, 480 (2011). source

      Not checked in the free run.

    • cites [22] DeLuca, D.S. et al. RNA-SeQC: RNA-seq metrics for quality control and process optimization. Bioinformatics 28, 1530–1532 (2012). source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The aberrant GC-content spike at 50% GC seen at ILM3 was not present in the fifth library prepared at an independent site, indicating that this spike arises from sample preparation.

    “In the case of GC distribution, the fifth library from each sample did not exhibit an aberrant spike at 50% GC-content like replicates 1–4 sequenced at the ILM3 site; it is likely that the aberrant spike is a result arising purely from sample preparation (Fig. 4a,b and Supplementary Fig. 18a), which confirms the primary source of variation put forward by both the GEUVADIS and SEQC Consortium”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [33] SEQC/MAQC-III Consortium. A comprehensive assessment of RNA-seq accuracy, reproducibility and information content by the Sequencing Quality Control Consortium. Nat. Biotechnol. 10.1038/nbt.2957 (24 August 2014).

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    PEER yielded higher accuracy than the original limma-voom method when validated against common intra-site DEGs, indicating that global normalization methods such as PEER can improve RNA-seq data across Illumina and PGM platforms.

    “Using the common intra-site DEGs to validate the called inter-site DEGs, the MCC41, 42 showed that PEER also yields a higher accuracy than the original limmavoom method (Supplementary Fig. 22d,e), indicating that global data normalization analysis methods such as PEER can also be used to improve RNA-seq data across both Illumina and the PGM platforms.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [41] Baldi, P., Brunak, S., Chauvin, Y., Andersen, C.A. & Nielsen, H. Assessing the accuracy of prediction algorithms for classification: an overview. Bioinformatics 16, 412–424 (2000). source

      Not checked in the free run.

    • cites [42] Shi, L. et al. The MicroArray Quality Control (MAQC)-II study of common practices for the development and validation of microarray-based predictive models. Nat. Biotechnol. 28, 827–838 (2010). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The higher average expression of the TaqMan-tested genes may affect normalization method performance for this reference set.

    “and this may affect normalization method performance for this reference set”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    GC content bias correction packages cqn and EDASeq could not remove the false positives, likely because GC content bias is not the only source of bias in gene expression data.

    “The application of GC content bias correction packages including cqn14 and EDASeq15 could not remove these false positives, likely because GC content bias is not the only source that contributes to bias in gene expression data.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [14] Hansen, K.D., Irizarry, R.A. & Wu, Z. Removing technical variability in RNA-seq data using conditional quantile normalization. Biostatistics 13, 204–216 (2012). source

      Not checked in the free run.

    • cites [15] Risso, D., Schwartz, K., Sherlock, G. & Dudoit, S. GC-content normalization for RNA-Seq data. BMC Bioinformatics 12, 480 (2011). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisenone given

    Given the globalization of research collaborations and the growing number of large RNA-seq cohorts, obtaining sequencing data across different institutes and platforms is inevitable.

    “With the globalization of research collaborations and the emergence of an increasing number of large RNA-seq cohorts, obtaining sequencing data across different institutes and platforms is inevitable.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    These metrics and internal controls complement those currently in use and provide additional resolution into the quality of an RNA-seq dataset.

    “These metrics and internal controls complement those currently in use and create additional resolution insights into the quality of an RNA-seq dataset”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Multiple testing was adjusted using the Benjamini and Hochberg method at a range of false discovery rates (0.05, 0.01 or 0.001).

    “We used the Benjamini and Hochberg adjustment for multiple testing at a variety of false discovery rates {FDR | 0.05 or 0.01 or 0.001}.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    With advanced data processing, even substantial bias could be corrected and value extracted from experiments combined across multiple laboratories.

    “given advanced data processing, even substantial bias could be corrected and value extracted from experiments combined from multiple laboratories”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    These best practices for quality control and analysis of RNA-seq data from different experiments or laboratories are of immediate relevance not only for large-scale RNA-seq studies but also for the analysis of smaller experiments in the context of other data.

    “These best practices for quality control and analysis of RNA-seq data from different experiments or laboratories can readily be implemented, and they are of immediate relevance not just for large-scale RNA-seq studies, but also the analysis of smaller experiments in the context of other data”
    Quote in the paperNot a substantial claim
32 of 32

  • The paper relies on this without support.

    Best practices for RNA-seq analysis are not fully defined, especially for data collected across multiple sequencing platforms or sites.

    “but best practices for analyzing RNA-seq data have not been fully defined, particularly for data collected with multiple sequencing platforms or at multiple sites.”

  • The paper relies on this without support.

    Before this study, no systematic examination existed of how site-specific bias affects detection of differentially expressed genes.

    “Yet, to date, there has been no systematic examination of the impact of site-specific bias in detecting differentially expressed genes (DEGs), which is often the primary goal of an RNA-seq experiment.”

  • The paper relies on this without support.

    The performance of competing methods for correcting site-specific biases had not been systematically characterized.

    “Moreover, there are various proposed means by which to correct for such biases, but the performance of several competing methods has not been systematically characterized.”

  • The paper relies on this without support.

    Identical RNA samples processed at different sites should produce little and random expression variation.

    “Ideally, expression values generated from identical samples at different sites should show little (and random) variation across sites.”

  • The paper relies on this without support.

    TaqMan measurements provide a reliable reference standard for true DEG status.

    “with the true positive rate (TPR) and false positive rate (FPR) based on the Taqman data set (Supplementary Fig. 11).”

  • The paper relies on this without support.

    This study is the first cross-platform evaluation of methods for assessing RNA-seq quality and removing variance for multi-site, multi-platform reproducibility.

    “To our knowledge, our work represents the first cross-platform evaluation of methods for assessing RNA-seq quality and removing variance from data for multi-site, multi-platform reproducibility, which is a prerequisite for reliable conclusions and the integration of measurements and experiments from different laboratories.”

  • The paper relies on this without support.

    The ENCODE project and the GEUVADIS Consortium have provided valuable guidelines and best practices for RNA-seq experiments.

    “The ENCODE project and GEUVADIS Consortium have provided extremely valuable guidelines and best practices for RNA-seq experiments”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • The claimed number of benchmark sites exceeds the described sites

    Minor

    The paper says it benchmarked the two platforms across twenty laboratory sites, but its experimental design describes six Illumina test sites and three PGM sites. Those platform-specific counts total nine test sites, not twenty.

  • The stated largest sample difference conflicts with the mixing design

    Minor

    The paper calls A versus D the comparison with the biggest expected difference. Under its stated mixture design, A versus B differs more than A versus D; the Discussion also identifies A versus B as the more biologically different comparison.

  • Sample D is assigned two different mixing ratios

    Minor

    The Results define D as a 1:3 mixture of A and B, but Online Methods define both C and D as 3:1 mixtures. This changes the stated composition of a sample used as a built-in control.

3 of 3

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Li, S., Łabaj, P. P., Zumbo, P., Sykacek, P., Shi, W., Shi, L., Phan, J., Wu, P.-Y., Wang, M., Wang, C., Thierry-Mieg, D., Thierry-Mieg, J., Kreil, D. P., & Mason, C. E.
Affiliations
  • Weill Cornell Medicine (US)
  • BOKU University (AT)
  • Cornell University (US)
  • Fudan University (CN)
  • Georgia Institute of Technology (US)
  • Loma Linda University (US)
  • University of Warwick (GB)
Corresponding author
  • Kreil, David P
  • Mason, Christopher E
ORCID
Venue
Nature Biotechnology
Publisher
Springer Science and Business Media LLC
Published
2014-08-24
Volume
32
Issue
9
Pages
888-895
ISSN
  • 0733-222X (ISSN-L)
  • 1087-0156
  • 1546-1696
DOI
10.1038/nbt.3000
PMC
PMC4160374
OpenAlex
W2030677229
Semantic Scholar
ac5e0caf4ff09abff6e4a51c1e1388dae7f7eaf8
Cited by
175
References
59
Citations by year
2026: 3 · 2025: 6 · 2024: 9 · 2023: 16 · 2022: 8 · 2021: 15 · 2020: 14 · 2019: 12 · 2018: 21 · 2017: 16 · 2016: 26 · 2015: 20 · 2014: 9
FWCI
9.70
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Biotechnology (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • Minerva Access (University of Melbourne) (submittedVersion)
Funders
  • Monique Weill-Caulier Trust
  • Vienna Science and Technology Fund
  • National Institutes of Health
  • Weill Cornell Medical College
  • National Human Genome Research Institute (R01HG006798)
  • National Cancer Institute (R01CA149566, U10CA180827)
  • National Institute of Neurological Disorders and Stroke (R01NS076465)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
Keywords
RNA-seq normalization, systematic bias, site-specific effects, reproducibility, differentially expressed genes, sequencing error rate, insert size, gene coverage, GC content, multi-site data integration, SVA, peers
MeSH terms
Quality Control, Reproducibility of Results, Sequence Analysis, RNA

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-05.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-05.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review