CoChat
Sign Up
Cited by 582Single-cell and spatial transcriptomicsGenomics and Phylogenetic StudiesGene expression and cancer classification

Power analysis of single-cell RNA-sequencing experiments

Open access
Svensson, Valentine, Natarajan, Kedar Nath, Ly, Lam-Ha, Miragaia, Ricardo J, Labalette, Charlotte, Macaulay, Iain C, Cvejic, Ana, Teichmann, Sarah A·2017·Nature Methods
DOI 10.1038/nmeth.4220PMC PMC5376499

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Single-cell RNA sequencing (scRNA-seq) has become an established and powerful method to investigate transcriptomic cell-to-cell variation, thereby revealing new cell types and providing insights into developmental processes and transcriptional stochasticity. A key question is how the variety of available protocols compare in terms of their ability to detect and accurately quantify gene expression. Here, we assessed the protocol sensitivity and accuracy of many published data sets, on the basis of spike-in standards and uniform data processing. For our workflow, we developed a flexible tool for counting the number of unique molecular identifiers (https://github.com/vals/umis/). We compared 15 protocols computationally and 4 protocols experimentally for batch-matched cell populations, in addition to investigating the effects of spike-in molecular degradation. Our analysis provides an integrated framework for comparing scRNA-seq protocols.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper uses ERCC/SIRV spike-ins and uniform processing to compare sensitivity and accuracy across 15 scRNA-seq protocols and 28 studies, plus batch-matched mESC experiments. It finds that scRNA-seq protocols vary widely in sensitivity but generally quantify expression accurately, with sequencing depth critical for sensitivity and less so for accuracy.

Key findings

  • Across 15 protocols, 28 studies, and 18,123 samples, scRNA-seq was more sensitive than bulk RNA-seq; detection limits ranged from 1 to 1,000 molecules per cell, and the top C1 protocols (CEL-Seq2, STRT-Seq, SMARTer) reached #M = 2, 3, and 4 at 1 million reads, while plate CEL-Seq2 had #M = 13.
  • Accuracy was generally high; the best protocols were SUPeR-Seq (R = 0.95) and CEL-Seq2 (R = 0.94). Accuracy saturated at 250,000 reads, whereas sensitivity saturated at about 4.5 million reads and increased by an order of magnitude from 100,000 to 1 million reads.
  • UMI counting did not perfectly achieve absolute quantification: the fitted molecular exponent c was systematically below 1, with a mode around 0.8, even for 10-bp UMIs; 4-bp UMIs averaged about 0.6.
  • Endogenous mRNA was captured more efficiently than ERCC spike-ins: CEL-Seq UMI counts were 5–10% of smFISH counts for 9 genes versus 0.5–1% of input for ERCCs; spike-in degradation was 19 ± 0.7% per freeze-thaw cycle for ERCCs and 18.5 ± 0.1% for SIRVs.

Methods

  • Retrospective analysis of published scRNA-seq datasets using ERCC spike-ins: 15 protocols, 28 studies, 18,123 samples, and approximately 30 × 10^9 sequencing reads; plus experimental batch-matched mESC comparisons of SMARTer, Smart-Seq2, and STRT-Seq on the C1 platform and a 10x Genomics Chromium dataset with ERCC/SIRV spike-ins.
  • Sensitivity per sample was estimated by logistic regression of spike-in detection versus input molecules, defining the detection limit at 50% detection probability; accuracy was measured by Pearson correlation between input spike-in concentration and measured expression; UMI efficiency was modeled as U = E·M and U = E·M^c.
  • Sequencing-depth effects were modeled with linear regression including a quadratic read-depth term and a protocol-specific performance parameter; spike-in degradation was modeled Bayesianly in Stan; UMI counting was implemented in the `umis` tool.

Limitations

  • Stated: the analysis relies on original authors’ accurate reporting of spike-in volumes and dilutions; spike-ins may not reflect endogenous mRNA capture because ERCCs have shorter poly(A) tails, lack a 5′ cap, and are not bound by mRNA-binding proteins; the spike-in molecule detection limit is a relative sensitivity measure, not an absolute one.
  • Stated: technical proficiency/lab skill effects cannot be completely ruled out; some published experiments may have degraded spike-ins; SIRV measurements were noisier due to mapping uncertainty, and SIRVs span only four abundance levels and are not compatible with sensitivity analysis.
  • Evident: the retrospective observational design means protocol rankings may be confounded by cell type, laboratory, batch, and other unmeasured factors despite modeling sequencing depth; the paper notes that a perfect comparison would require multiple laboratories using a single reagent stock and mRNA dilution ladders.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/nmeth.4220

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 6 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Conventional bulk RNA-sequencing has higher accuracy than scRNA-seq protocols.

    “The accuracy of conventional bulk RNA-sequencing is higher than scRNA-seq protocols.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Sensitivity saturates at about 4.5 million reads per sample.

    “Based on the model, we find that sensitivity saturates at about 4.5 million reads per sample.”
    Quote in the paperSubstantial claim
  • Findingown data

    In the same samples, average ERCC spike-in UMI counts correspond to 0.5-1% of input molecule counts.

    “This clearly contrasts with the ERCC spike-ins in the same samples, where the average UMI counts correspond to 0.5-1% of input molecule counts (Supplementary Figure 2E).”
    Quote in the paperSubstantial claim
  • Findingown data

    When the UMI model is extended to U=E·Mc, the best-fit molecular exponent c is systematically below 1, with a mode around 0.8.

    “Instead, we find that the best fit is systematically lower than 1, with a mode of around 0.8 (Supplementary Figure 2C).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: For UMI-tag counting, the number of UMIs for a gene equals the UMI efficiency multiplied by the number of RNA molecules of that gene.
  • Findingown data

    Increasing read depth from 100,000 to 1 million reads per sample increases sensitivity by an order of magnitude.

    “However the increase from 100,000 reads to 1 million reads per sample results in increased sensitivity of an order of magnitude.”
    Quote in the paperSubstantial claim
  • May combine two claims. 37
  • Classified as probably not a substantial claim. 13

  • May combine two claims.

    Findingown data

    The analysis covers 15 distinct experimental protocols and 28 single-cell studies, including 17 whole-transcript coverage studies and 11 UMI-based studies.

    “Our analysis spans 15 distinct experimental protocols encompassing 28 single-cell studies, including 17 studies with traditional whole-transcript coverage based strategy for measuring expression levels and 11 that used strategies based on UMI’s for digital quantification of transcripts (Supplementary Table 1).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The three C1 microfluidics protocols, CEL-Seq2, STRT-Seq, and SMARTer, were top performers in molecule detection, with #M values of 2, 3, and 4 respectively.

    “The three protocols implemented in a C1 microfluidics system (CEL-Seq2 (C1), STRT-Seq (C1), and SMARTer (C1); number of molecules at a million reads,#M = 2,3,and4respectively) were the top performing protocols in terms of molecule detection.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Logistic regression across all samples shows that scRNA-seq protocols are more sensitive than bulk RNA-sequencing and can detect very low numbers of input molecules.

    “Applying logistic regression to all samples, we observe that scRNA-seq protocols are more sensitive than bulk RNA-sequencing and can detect very low numbers of input molecules (Figure 2B).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    Sequencing depth is a technical feature, and the number of genes detected depends on depth.

    “It is equally important to note that sequencing depth is a technical feature, and the number of genes detected depend on the depth.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The best-performing protocols in accuracy are SUPeR-Seq (R=0.95) and CEL-Seq2 (R=0.94).

    “We found that the best performing protocols in terms of accuracy are SUPeR-Seq (R=0.95), the only total-RNA protocol for single cells to date, and CEL-Seq2 (R=0.94), which uses In Vitro Transcription (IVT) rather than PCR to amplify cDNA.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Technical sensitivity is critically dependent on sequencing depth, and sensitivity comparisons that ignore depth differences would be misleading.

    “We find that the technical sensitivity is critically dependent on sequencing depth, and sensitivity comparisons without accounting for differences in depth would be misleading (Figure 3B).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Even in protocols with 10-base-pair UMIs, the molecular exponent remains about 0.8 per sample on average and rarely reaches 1.

    “However, even in protocols with longer UMI’s of 10 base pairs (which can count over a million unique molecules), the molecular exponent is still 0.8 per sample on average, and rarely reaches 1.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The analysis includes 18,123 publicly available samples and 30x10^9 sequencing reads.

    “In total, our analysis consists of 18,123 publicly available samples of 30x109 sequencing reads.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The accuracy of scRNA-seq protocols is still high, and individual samples rarely have a Pearson correlation below 0.6.

    “Remarkably, the accuracy of scRNA-seq protocols is still high, and rarely do individual samples have a Pearson correlation lower than 0.6.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The UMI efficiency model recapitulates the logistic-regression sensitivity measure, with high-efficiency samples having low molecule detection limits, except for MARS-Seq.

    “The results recapitulate the logistic regression based measure for sensitivity, as samples with high efficiency have a low molecule detection limit (with the exception of MARS-Seq data, Supplementary Figure 2B).”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: For UMI-tag counting, the number of UMIs for a gene equals the UMI efficiency multiplied by the number of RNA molecules of that gene.

  • May combine two claims.

    Findingown data

    UMIs of length 4 base pairs can count up to 256 unique molecules and have an average molecular exponent of 0.6.

    “For example, UMIs with length of 4 base pairs can only count up to 256 unique molecules, and have on average a molecular exponent of 0.6.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The empty median confidence interval is 100% contained within the non-empty median confidence interval for accuracy and 84% for sensitivity.

    “We found that empty median CI is 100% contained within the non-empty median CI for accuracy, and 84% for sensitivity.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    STRT-Seq samples had very poor accuracy because 5' transcript tags alone cannot distinguish mRNA isoforms.

    “The STRT-Seq samples had very poor accuracy, as expected since the 5’ transcript tags alone cannot distinguish between different mRNA isoforms.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Freeze-thaw cycles had an overall small effect on accuracy, and this effect was similar between protocols.

    “The freeze-thaw cycles emulate normal handling and comparing these to our in-house protocols, we observed that accuracy metric had an overall small effect and was similar between protocols (Figure 4A).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Repeated freeze-thaw cycles emulate normal reagent handling.

  • May combine two claims.

    Findingown data

    The SIRV degradation rate is 18.5±0.1%, recapitulating the ERCC degradation rate.

    “We also applied the mRNA degradation model to SIRV’s, and found a degradation rate of 18.5±0.1%, which recapitulates ERCC degradation rates.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The gold-standard annotation of the spike-ins is correct.

  • May combine two claims.

    Findingown data

    Extreme overnight degradation at room temperature or 37°C causes two orders of magnitude difference in sensitivity compared with other samples, similar to differences between protocols.

    “The sensitivity metric for samples subjected to conditions as extreme as overnight degradation (room temperature or 37°C) had two orders of magnitude difference compared to other samples, similar to the difference between protocols (Figure 4A).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Although normal handling did not cause large variation in molecular detection limit or accuracy, the relative abundance of spike-ins in a sample is strongly affected by freeze-thaw cycles.

    “While we did not observe a large variation in molecular detection limit or accuracy due to normal handling, the relative abundance of spike-ins in a sampling is strongly affected by freeze-thaw cycles.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Repeated freeze-thaw cycles emulate normal reagent handling.

  • May combine two claims.

    Findingreasoning

    Both sensitivity and accuracy depend closely on the specific scRNA-seq protocol used to generate the data.

    “However, both sensitivity and accuracy are closely dependent on the scRNA-seq protocol used to generate the data.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ERCC spike-ins serve as adequate proxies for cellular mRNA when evaluating sensitivity and accuracy.

  • May combine two claims.

    Findingown data

    Accuracy and sensitivity are equivalent between empty and non-empty samples from three studies, confirming that endogenous mRNA content does not affect the performance metrics.

    “We compared accuracy and sensitivity between empty and non-empty samples from three different studies and found equivalent results, confirming that endogenous mRNA content does not affect the performance metrics (Figure 4C).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The ERCC spike-in collection contains 92 RNA species of different lengths and GC content, mixed at known concentrations, and covers 22 abundance levels separated by one fold change.

    “The standardized ERCC (External RNA Controls Consortium)5 spike-in collections consist of a set of 92 RNA molecule species of varying lengths and GC contents, mixed at known concentrations, and represent 22 abundance levels that are spaced one fold change apart from each other (Supplementary Figure 1).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] External RNA Controls Consortium. Proposed methods for testing and selecting the ERCC external RNA controls. BMC Genomics 6, 150 (2005). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Spike-in molecules may not fully reflect endogenous mRNA capture efficiency in scRNA-seq because they differ from natural mRNA in features such as polyA tail length and binding by proteins.

    “In addition, spike-in molecules may not truly reflect endogenous mRNA capture efficiency in scRNA-seq owing to deviation from natural mRNA sequence features such as shorter polyA tails and mRNA binding proteins.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Lower accuracy and variable Pearson correlations among individual cells in GnT-Seq, CEL-Seq, and MARS-Seq may indicate variable protocol success rates.

    “The lower accuracy and variable Pearson correlations for individual cells (samples) within protocols (GnT-Seq, CEL-Seq, MARS-Seq), might be indicative of variable success rates of protocols.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Although UMIs should remove amplification biases, the assumed absolute quantification does not hold perfectly.

    “In conclusion, while UMIs should provide a way of removing amplification biases, the assumed absolute quantification does not seem to hold perfectly.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The accuracy metric is based on relative abundances and is not affected by the difference in capture efficiency.

    “The accuracy metric is based on relative abundances, and is not affected by this.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The sensitivity estimates based on spike-ins are conservative and likely underestimate rather than overestimate.

    “Overall, this comparison implies that our sensitivity estimates are conservative, and are likely to underestimate rather than overestimate.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    Endogenous mRNA have complex secondary structures and can be bound by proteins, which may reduce reverse transcription efficiency.

    “On the other hand, endogenous mRNA have intricate secondary structures and can be bound to proteins, potentially reducing the efficiency of reverse transcription.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    For detected mRNAs in scRNA-seq, expression levels are generally accurate and quantitatively meaningful.

    “This also suggests that for detected mRNAs in scRNA-seq, the expression levels are generally accurate and quantitatively meaningful.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    ERCC spike-ins are not capped at the 5' end, which may reduce template-switching efficiency compared with endogenous mRNAs.

    “Furthermore, the ERCC spike-ins are not capped at the 5’ end, which may further lead to reduced template switching efficiency (used in several protocols) as compared to endogenous mRNAs14.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] Kapteyn, J., He, R., McDowell, E.T. & Gang, D.R. Incorporation of non-natural nucleotides into template-switching oligonucleotides reduces background and improves cDNA synthesis from very small RNA samples. BMC Genomics 11, 413 (2010). source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Because the degradation rate is conserved between ERCC and SIRV spike-ins, the approximately 20% degradation rate per freeze-thaw cycle may hold for RNA in general.

    “As we also found that the degradation rate was conserved between ERCC and SIRV spike-ins, the approximately 20% degradation rate per freeze-thaw cycle may hold for RNA in general.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Comparison of smFISH values with calculated spike-in molecules per cell suggests that endogenous RNA is captured and amplified about one order of magnitude more efficiently than spike-ins, and that protocols are less sensitive than actual.

    “This comparison also suggests more efficient capture and amplification of endogenous RNA than spike-ins by about one order of magnitude, and lower sensitivity of protocols than actual.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    ERCC spike-ins have short poly-A tails of 20 to 26 bases, mostly 24 bases, whereas eukaryotic mRNAs have poly-A tails of about 250 bases.

    “The ERCC spike-ins have short poly-A tails ranging from 20 to 26 bases long (the majority are 24 bases) in comparison to eukaryotic mRNAs of 250 base long poly-A tails10.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10] Viphakone, N., Voisinet-Hakil, F. & Minvielle-Sebastia, L. Molecular dissection of mRNA poly(A) tail length control in yeast. Nucleic Acids Res. 36, 2418–2433 (2008). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The logistic regression analysis included only samples with at least eight detected spike-ins.

    “The logistic regression analysis was limited to samples with at least eight spike-ins detected.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Poly-T priming of ERCC spike-ins might be less efficient than for endogenous mRNA, possibly making all protocols appear less sensitive than they actually are for endogenous mRNAs.

    “This suggests that poly-T priming of ERCC spike-ins might be less efficient than for endogenous mRNA, possibly making all protocols seem less sensitive than they actually are for endogenous mRNAs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Thirty samples were excluded because they were annotated as lacking added ERCC spike-ins.

    “Thirty samples were excluded as they were annotated as not having had ERCC spike-ins added to them.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    SIRV measurements were more noisy, likely because of mapping uncertainty.

    “However the SIRVs measurements were more noisy, likely due to mapping uncertainty (Also mentioned in Discussion).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Unlike endogenous mRNAs, spike-in RNA are not naturally bound by mRNA-binding proteins or have secondary structures.

    “Lastly, unlike endogenous mRNAs, spike-in RNA are not naturally bound by mRNA-binding proteins or have secondary structures.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Spike-in RNA are not naturally bound by mRNA-binding proteins or have secondary structures.

  • May combine two claims.

    Interpretationown data

    Accuracy calculated from SIRVs recapitulates accuracy from ERCCs, indicating that spike-in batch variability does not generally explain differences between protocols.

    “Comparing the protocols, accuracy calculated based on SIRV’s recapitulates accuracy based on ERCC’s, indicating that spike-in batch variability does not in general explain differences between protocols.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    A protocol that performs poorly in this comparison is not necessarily unsuitable, and the same protocol could achieve better performance.

    “A protocol showing poor performance in our comparison does not restrict its suitability and it is indeed possible to get better performance for the same protocol.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    When fewer than eight spike-ins were detected, accuracy was set to infinity because this was deemed insufficient evidence to estimate accuracy.

    “If less than 8 spike-ins were observed, the accuracy was set to infinity, as we consider this to be insufficient evidence to estimate the accuracy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    Some published experiments may have used substantially degraded spike-ins, which could affect the performance metrics.

    “Nevertheless, one needs to be aware that in certain published experiments, spike-ins may have been greatly degraded with an impact on our performance metrics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A previous study showed that read alignment to ERCC spike-ins varies widely between libraries and platforms, with some spike-ins having reproducibly poor behavior.

    “A previous study showed13 that read alignment to ERCC spike-ins varies widely between libraries and platforms, with some spike-ins having reproducibly poor behavior.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] SEQC/MAQC-III Consortium. A comprehensive assessment of RNA-seq accuracy, reproducibility and information content by the Sequencing Quality Control Consortium. Nat. Biotechnol. 32, 903–914 (2014). source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Miniaturized reaction volumes in scRNA-seq protocols increase sensitivity and yield good value when sequencing approximately one million reads per sample.

    “Overall, our comparison suggests that miniaturized reaction volumes in scRNA-seq protocols increases sensitivity and provides a good return on investment when sequencing around a million reads per sample.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ERCC spike-ins serve as adequate proxies for cellular mRNA when evaluating sensitivity and accuracy.
42 of 42

  • The authors state this as an assumption.

    The original authors accurately reported spike-in volumes and dilution.

    “We rely on accurate reporting of spike-in volumes and dilution by original authors, and have reconfirmed in few cases (personal communication).”

  • The authors state this as an assumption.

    For UMI-tag counting, the number of UMIs for a gene equals the UMI efficiency multiplied by the number of RNA molecules of that gene.

    “If E is the UMI (counting) efficiency, the underlying assumption is that the number of UMIs of a gene U = E·M, where 0 < E < 1 (Supplementary Figure 2A) and M is the number of RNA molecules of a gene.”

  • The authors state this as an assumption.

    The gold-standard annotation of the spike-ins is correct.

    “The analyzed performance metrics of the scRNA-seq protocols inherently assumes the gold standard annotation of the spike-ins to be correct.”

  • The paper relies on this without support.

    All ERCC spike-ins within a sample are equally affected by capture and amplification biases.

    “The accuracy metric remains unaffected by these issues, as all ERCC spike-ins within a sample are equally affected.”

  • The authors state this as an assumption.

    Repeated freeze-thaw cycles emulate normal reagent handling.

    “The freeze-thaw cycles emulate normal handling and comparing these to our in-house protocols, we observed that accuracy metric had an overall small effect and was similar between protocols (Figure 4A).”

  • The paper relies on this without support.

    Spike-in RNA are not naturally bound by mRNA-binding proteins or have secondary structures.

    “Lastly, unlike endogenous mRNAs, spike-in RNA are not naturally bound by mRNA-binding proteins or have secondary structures.”

  • The paper relies on this without support.

    ERCC spike-ins serve as adequate proxies for cellular mRNA when evaluating sensitivity and accuracy.

    “A control ladder of mRNA would eliminate issues arising from differences between synthetic spike-ins and mRNA.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • These challenges apply to bulk RNA-seq workflows but are exacerbated due to the extreme comparative sparsity of scRNA-seq data.

    “These same challenges apply to bulk RNA-seq workflows, but are exacerbated due to the extreme comparative sparsity of scRNA-seq data [5].”

    From Normalization and variance stabilization of single-cell RNA-seq data using regularized negative binomial regression (2019) · cites this paper as [5]

    The Introduction says scRNA-seq starts with tiny amounts of material and requires amplification, making detection sensitivity an important challenge. The cited passages do not establish that scRNA-seq data are comparatively sparser than bulk RNA-seq data or that shared workflow challenges are exacerbated by sparsity; the Results instead report greater molecular detection sensitivity for the scRNA-seq protocols studied. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • The capture efficiency of molecules in droplet-based experiments varies by as much as an order of magnitude.

    “This is likely a consequence of the fact that the capture efficiency of molecules in droplet-based experiments varies by as much as an order of magnitude [17].”

    From SoupX removes ambient RNA contamination from droplet-based single-cell RNA sequencing data (2020) · cites this paper as [17]

    The Results section on technical sensitivity reports substantial variation across scRNA-seq protocols, including a droplet-based protocol, and high within-protocol variability. But it does not establish that molecular capture efficiency in droplet-based experiments varies by an order of magnitude. The Discussion’s one-order-of-magnitude figure compares endogenous RNA with spike-ins, not variation among droplet experiments; page numbers were unavailable in our copy. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • In the Chromium control dataset, identical patterns to the PBMC dataset were observed, including ineffectual normalization of high-abundance genes.

    “when we analyzed a 10X Chromium dataset that used human brain RNA as a control (“Chromium control dataset” [5]), we observed identical patterns, and in particular, ineffectual normalization of high-abundance genes”

    From Normalization and variance stabilization of single-cell RNA-seq data using regularized negative binomial regression (2019) · cites this paper as [5]

  • Venus protein expression correlates mostly with RNA findings, except weak acinar expression around E18.5 which appears underestimated in scRNA sequencing data.

    “These patterns correlate mostly with the RNA findings, except the weak but robust acinar expression around E18.5 which appears underestimated in scRNA sequencing data potentially attributed to technical limitations for weakly expressed genes [50–54].”

    From TBX3 is dynamically expressed in pancreatic organogenesis and fine-tunes regeneration (2023) · cites this paper as [50]

  • Sensitivity and accuracy of scRNA-seq data were estimated using ERCC spike-ins with known concentrations and a published algorithm.

    “We used a previously published algorithm to estimate sensitivity and accuracy of scRNA-seq data based on ERCC spike-ins with known concentrations (Svensson et al., 2017).”

    From Developmental Heterogeneity of Microglia and Brain Myeloid Cells Revealed by Deep Single-Cell RNA Sequencing (2019) · cites this paper as Svensson et al., 2017

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Svensson, V., Natarajan, K. N., Ly, L.-H., Miragaia, R. J., Labalette, C., Macaulay, I. C., Cvejic, A., & Teichmann, S. A.
Affiliations
  • European Bioinformatics Institute (GB)
  • Wellcome Sanger Institute (GB)
  • Wellcome/MRC Cambridge Stem Cell Institute (GB)
Corresponding author
Svensson, Valentine
ORCID
Venue
Nature Methods
Publisher
Springer Science and Business Media LLC
Published
2017-03-06
Volume
14
Issue
4
Pages
381-387
ISSN
  • 1548-7091 (ISSN-L)
  • 1548-7105
DOI
10.1038/nmeth.4220
PMC
PMC5376499
OpenAlex
W2591733518
Semantic Scholar
893d91b4a527865c25d64abaedd2dfb5008aacd7
Cited by
568
References
45
Citations by year
2026: 29 · 2025: 35 · 2024: 47 · 2023: 54 · 2022: 57 · 2021: 70 · 2020: 69 · 2019: 79 · 2018: 78 · 2017: 49 · 2015: 1
FWCI
30.70
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Methods (publishedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
  • Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion)
  • Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, cc-by-nc-nd)
  • Apollo (University of Cambridge) (submittedVersion)
  • Apollo
Funders
  • Wellcome Trust
  • Cancer Research UK (14953)
  • Medical Research Council (MC_PC_12009)
  • Biotechnology and Biological Sciences Research Council (BBS/E/T/000PR9819)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
Keywords
single-cell RNA sequencing, power analysis, unique molecular identifiers, molecular degradation, gene expression quantification
MeSH terms
Animals, Freezing, Poly A, RNA, Messenger, Sensitivity and Specificity, Sequence Analysis, RNA, Mice, Embryonic Stem Cells, Workflow, Single-Cell Analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review