PreprintCited by 1,210Single-cell and spatial transcriptomicsGene expression and cancer classificationBioinformatics and Genomic Networks

The DOI matches its Crossref record; 105 of the 105 references it cites were found in a registry or an index; 54 of the 64 sources its claims cite were checked for support.

Confronting false discoveries in single-cell differential expression

Open access

Abstract

Differential expression analysis in single-cell transcriptomics enables the dissection of cell-type-specific responses to perturbations such as disease, trauma, or experimental manipulation. While many statistical methods are available to identify differentially expressed genes, the principles that distinguish these methods and their performance remain unclear. Here, we show that the relative performance of these methods is contingent on their ability to account for variation between biological replicates. Methods that ignore this inevitable variation are biased and prone to false discoveries. Indeed, the most widely used methods can discover hundreds of differentially expressed genes in the absence of biological differences. Our results suggest an urgent need for a paradigm shift in the methods used to perform differential expression analysis in single-cell data.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The paper benchmarks 14 differential expression methods on 18 ground-truth datasets and finds that pseudobulk methods, which aggregate cells within biological replicates, outperform single-cell methods. Methods that ignore between-replicate variation are biased toward highly expressed genes and can produce hundreds of false discoveries even without biological differences.

Key findings

  • In 18 ground-truth datasets, all six top-performing DE methods aggregated cells within biological replicates to form pseudobulks; pseudobulk versus single-cell differences were highly significant and robust to concordance methodology.
  • Single-cell DE methods were biased toward highly expressed genes: false positives were more highly expressed, abundant ERCC spike-ins were incorrectly called DE, and a compendium of 46 scRNA-seq datasets showed a systematic preference for highly expressed genes.
  • Failing to account for biological replicates produced false discoveries: simulations identified hundreds of DE genes with no perturbation; random splits of unstimulated controls in 14 datasets produced excess false positives; spatial transcriptomics identified thousands of DE genes within each spinal cord region.
  • In injured mouse spinal cord snRNA-seq (19,237 nuclei), RNAscope validated 5 of 6 genes called by edgeR-LRT versus 3 of 13 called by Wilcoxon rank-sum (p < 0.05, χ² test); GLMMs converged to pseudobulk performance in datasets of 500 or more cells and took an average of 13.5 h per cell type versus minutes for pseudobulk methods.

Methods

  • Literature review of 500 recent scRNA-seq studies, of which 205 conducted DE analysis between biological conditions, identified 14 methods: 7 single-cell methods, 6 pseudobulk methods, and 1 linear mixed model.
  • The study used 18 ground-truth datasets from four publications with matched bulk and scRNA-seq in the same purified cells, perturbations, and laboratories; concordance was measured by area under the concordance curve (AUCC) and rank correlation, with GO term enrichment, a 46-dataset scRNA-seq compendium, Splatter simulations, 14 control-group datasets, spatial transcriptomics, and spinal cord injury snRNA-seq plus RNAscope.
  • Additional analyses disabled pseudobulk aggregation, created pseudo-replicates by shuffling replicates, compared variance between biological replicates and pseudo-replicates, and evaluated Poisson or negative binomial GLMMs.

Limitations

  • The paper does not include a dedicated limitations section. It states that the enormous computational requirements of GLMMs prevented evaluation in the full ground-truth datasets, so downsampled datasets of 25–1,000 cells were used; the ground-truth resource was limited to 18 datasets from four publications.
  • Evident from the text: simulations used parameters estimated from the Kang et al. dataset and may not capture all biological or technical heterogeneity; the in vivo RNAscope validation screened selected putatively DE genes rather than all DE genes; and human-versus-mouse variability conclusions rely on the assembled compendium datasets.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1101/2021.03.12.435024

Cites

105 of 105 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

54 of 64 cited sources checked · 24 supported · 2 flagged · accuracy not yet measured

Show all 138 claims

  • It may state two separate claims as one. 21
  • Probably not a substantial claim. 21
  • The quote only partly states this claim. 2
40 of 40
  • The authors state this as an assumption.

    The closest possible approximation to the experimental ground truth can be obtained from matched bulk and scRNA-seq performed on the same population of purified cells, exposed to the same perturbations, and sequenced in the same laboratories.

    “We posited that the closest possible approximation to this ground truth could be obtained from matched bulk and scRNA-seq performed on the same population of purified cells, exposed to the same perturbations, and sequenced in the same laboratories.”
  • The paper relies on this without support.

    Measurements of gene expression in single cells are inherently sparse.

    “Measurements of gene expression in single cells are inherently sparse.”
  • The paper relies on this without support.

    Many factors that produce variability between biological replicates can be minimized in animal models, including genetic background, environment, intensity and timing of the biological perturbation, and sample processing.

    “Many of the factors that produce this variability between replicates can be minimized in animal models, including the genetic background, environment, intensity and timing of the biological perturbation, and sample processing.”
  • The paper relies on this without support.

    Sources of variation between biological replicates are inherently more difficult to control in experiments involving human subjects.

    “In contrast, these sources of variation are inherently more difficult to control in experiments involving human subjects.”
  • The paper relies on this without support.

    The pervasiveness of false discoveries in published analyses of single-cell data and the propensity for these false discoveries to affect the biological conclusions of a study have remained unclear.

    “Consequently, the pervasiveness of false discoveries in published analyses of single-cell data and the propensity for these false discoveries to affect the biological conclusions of a study have remained unclear.”
  • The paper relies on this without support.

    The 500 most recent entries in the curated database are representative of scRNA-seq studies published in the past two years.

    “we limited our analysis to the 500 most recently published studies”
  • The paper relies on this without support.

    The 18 ground truth datasets are representative of single-cell experiments for assessing DE methods.

    “which led us to compile a resource of eighteen ground truth datasets from four publications12–15”
  • The paper relies on this without support.

    The 46 published scRNA-seq datasets in the compendium are representative of single-cell RNA-seq studies generally.

    “We assembled a compendium of 46 published single-cell or single-nucleus RNA-seq studies (Supplementary Fig. 3), and performed DE analyses across this compendium to establish the generality of our conclusions.”
  • The paper relies on this without support.

    Randomly splitting a control group into artificial control and treatment groups creates a valid null distribution for DE analysis.

    “For each of these studies, we split the control group randomly into artificial ‘control’ and ‘treatment’ groups, and performed DE analysis.”
9 of 9

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Human and mouse dataset count differs between Results and Methods

    Minor

    The Results say variability was assessed in 41 human and mouse datasets, whereas the Methods specify 38 datasets, comprising 18 human and 20 mouse datasets.

  • Number of edgeR-LRT RNAscope probes conflicts with the validation count

    Minor

    The Methods claim probes were obtained for seven edgeR-LRT genes but name only six. The Results also describe six edgeR-LRT genes in the RNAscope validation.

2 of 2

Citations

Cites

Flagged first, then by citations

Show all 104 references

Research works printed but not linked: 1

  1. 10.1007/978-1-4939-3578-9$\_$19 · DOI

Cited by

Sorted by citations

Items 101–200 of 1,210

Metadata

Authors
Squair, J. W., Gautier, M., Kathe, C., Anderson, M. A., James, N. D., Hutson, T. H., Hudelle, R., Qaiser, T., Matson, K. J. E., Barraud, Q., Levine, A. J., La Manno, G., Skinnider, M. A., & Courtine, G.
Affiliations
  • University of British Columbia (CA)
  • Centre Hospitalier Universitaire Vaudois (CH)
  • National Institute of Neurological Disorders and Stroke (US)
Corresponding author
  • Skinnider, Michael A.
  • Courtine, Grégoire
Venue
Nature Communications
Publisher
openRxiv
Published
2021-03-12
Volume
12
Issue
1
Pages
5692-5692
ISSN
2041-1723 (ISSN-L)
OpenAlex
W3202649987
Cited by
1,086
References
104
Citations by year
2026: 396 · 2025: 214 · 2024: 231 · 2023: 145 · 2022: 86 · 2021: 12
FWCI
34.62
Type
journal article
Language
en
Open access
Locations
  • Nature Communications (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (submittedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • SERVAL (Université de Lausanne) (publishedVersion, cc-by)
  • IRIS (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
Funders
  • National Science Foundation
  • Wings for Life
  • Compute Canada
  • Western Canada Research Grid
  • Wyss Center for Bio and Neuroengineering
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (185728, 192558, 193445, 190495, 310030)
  • University of British Columbia
  • National Institutes of Health
  • Canadian Institutes of Health Research
  • National Supercomputing Center, Korea Institute of Science and Technology Information
  • National Institute of Neurological Disorders and Stroke
  • H2020 European Research Council (682999)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
Keywords
single-cell differential expression, false discovery, biological replicates, comparison of statistical methods, differentially expressed genes, spinal cord injury, mouse spinal cord, cell type-specific responses, false positives
MeSH terms
Data Accuracy, Biological Variation, Population, Biological Variation, Individual, RNA-Seq, Animals, Gene Expression Regulation, Humans, Rabbits, Swine, Models, Statistical, Mice, Rats, Single-Cell Analysis, Datasets as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review