Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 107 of the 107 references it cites were found in a registry or an index; 54 of the 64 sources its claims cite were checked for support.

Confronting false discoveries in single-cell differential expression

Open access

Abstract

Differential expression analysis in single-cell transcriptomics enables the dissection of cell-type-specific responses to perturbations such as disease, trauma, or experimental manipulation. While many statistical methods are available to identify differentially expressed genes, the principles that distinguish these methods and their performance remain unclear. Here, we show that the relative performance of these methods is contingent on their ability to account for variation between biological replicates. Methods that ignore this inevitable variation are biased and prone to false discoveries. Indeed, the most widely used methods can discover hundreds of differentially expressed genes in the absence of biological differences. Our results suggest an urgent need for a paradigm shift in the methods used to perform differential expression analysis in single-cell data.

Overview

CoChat Summary

TL;DR

The paper benchmarks 14 differential expression methods on 18 ground-truth datasets and finds that pseudobulk methods, which aggregate cells within biological replicates, outperform single-cell methods. Methods that ignore between-replicate variation are biased toward highly expressed genes and can produce hundreds of false discoveries even without biological differences.

Key findings

  • In 18 ground-truth datasets, all six top-performing DE methods aggregated cells within biological replicates to form pseudobulks; pseudobulk versus single-cell differences were highly significant and robust to concordance methodology.
  • Single-cell DE methods were biased toward highly expressed genes: false positives were more highly expressed, abundant ERCC spike-ins were incorrectly called DE, and a compendium of 46 scRNA-seq datasets showed a systematic preference for highly expressed genes.
  • Failing to account for biological replicates produced false discoveries: simulations identified hundreds of DE genes with no perturbation; random splits of unstimulated controls in 14 datasets produced excess false positives; spatial transcriptomics identified thousands of DE genes within each spinal cord region.
  • In injured mouse spinal cord snRNA-seq (19,237 nuclei), RNAscope validated 5 of 6 genes called by edgeR-LRT versus 3 of 13 called by Wilcoxon rank-sum (p < 0.05, χ² test); GLMMs converged to pseudobulk performance in datasets of 500 or more cells and took an average of 13.5 h per cell type versus minutes for pseudobulk methods.

Methods

  • Literature review of 500 recent scRNA-seq studies, of which 205 conducted DE analysis between biological conditions, identified 14 methods: 7 single-cell methods, 6 pseudobulk methods, and 1 linear mixed model.
  • The study used 18 ground-truth datasets from four publications with matched bulk and scRNA-seq in the same purified cells, perturbations, and laboratories; concordance was measured by area under the concordance curve (AUCC) and rank correlation, with GO term enrichment, a 46-dataset scRNA-seq compendium, Splatter simulations, 14 control-group datasets, spatial transcriptomics, and spinal cord injury snRNA-seq plus RNAscope.
  • Additional analyses disabled pseudobulk aggregation, created pseudo-replicates by shuffling replicates, compared variance between biological replicates and pseudo-replicates, and evaluated Poisson or negative binomial GLMMs.

Limitations

  • The paper does not include a dedicated limitations section. It states that the enormous computational requirements of GLMMs prevented evaluation in the full ground-truth datasets, so downsampled datasets of 25–1,000 cells were used; the ground-truth resource was limited to 18 datasets from four publications.
  • Evident from the text: simulations used parameters estimated from the Kang et al. dataset and may not capture all biological or technical heterogeneity; the in vivo RNAscope validation screened selected putatively DE genes rather than all DE genes; and human-versus-mouse variability conclusions rely on the assembled compendium datasets.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1101/2021.03.12.435024

Cites

107 of 107 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 A systematic benchmark of differential expression in single-cell transcriptomics. a Schematic overview of the eighteen ground-truth datasets analyzed in this study. b Statistical methods for DE analysis employed in 500 recent scRNA-seq papers. Grey bars represent DE analysis methods included in this study. “Other” includes methods used in two or fewer studies. Inset pie chart shows the total proportion of recent scRNA-seq papers that employed DE analysis methods included in this study. Source data are provided as a Source Data file. c Area under the concordance curve (AUCC) for fourteen DE methods in the eighteen ground-truth datasets shown in a. d Mean difference in the AUCC (∆ AUCC) between the fourteen DE methods shown in c. Asterisks indicate comparisons with a two-tailed t-test p-value less than 0.05. e AUCC of GO term enrichment, as evaluated using gene set enrichment analysis46, in the eighteen ground-truth datasets shown in a. f Rank and statistical significance of the GO term GO:0043330 (“response to exogenous dsRNA”) in GSEA analyses of mouse bone marrow mononuclear cells stimulated with poly-I:C, a type of synthetic dsRNA, for four h, using the output of fourteen DE methods. Source data are provided as a Source Data file.
Fig. 1 A systematic benchmark of differential expression in single-cell transcriptomics. a Schematic overview of the eighteen ground-truth datasets analyzed in this study. b Statistical methods for DE analysis employed in 500 recent scRNA-seq papers. Grey bars represent DE analysis methods included in this study. “Other” includes methods used in two or fewer studies. Inset pie chart shows the total proportion of recent scRNA-seq papers that employed DE analysis methods included in this study. Source data are provided as a Source Data file. c Area under the concordance curve (AUCC) for fourteen DE methods in the eighteen ground-truth datasets shown in a. d Mean difference in the AUCC (∆ AUCC) between the fourteen DE methods shown in c. Asterisks indicate comparisons with a two-tailed t-test p-value less than 0.05. e AUCC of GO term enrichment, as evaluated using gene set enrichment analysis46, in the eighteen ground-truth datasets shown in a. f Rank and statistical significance of the GO term GO:0043330 (“response to exogenous dsRNA”) in GSEA analyses of mouse bone marrow mononuclear cells stimulated with poly-I:C, a type of synthetic dsRNA, for four h, using the output of fourteen DE methods. Source data are provided as a Source Data file.CC BY · PubMed Central
Fig. 2 Single-cell DE methods are biased towards highly expressed genes. a Schematic illustration of the creation of ‘pseudobulks’ from single-cell data. Top, biological replicate from which each cell was obtained. Bottom, simulated gene expression matrix. Read counts for each gene are aggregated across cells of a given type within each biological replicate. b Mean AUCCs across eighteen ground-truth datasets after dividing the transcriptome into terciles of lowly, moderately, or highly expressed genes. c Mean expression levels of the 100 top-ranked false-positive genes from each DE method. d Spearman correlation between the mean expression of 80 ERCC spike-ins expressed in at least three cells and the –log10 p-value of differential expression assigned by each DE method. e Scatterplots of mean ERCC expression vs. –log10 p-value for exemplary single-cell and pseudobulk DE methods. Trend lines and shaded areas show local polynomial regression and the 95% confidence interval, respectively. f Mean expression levels of the 200 top-ranked genes from each DE method in a collection of 46 scRNA-seq datasets.
Fig. 2 Single-cell DE methods are biased towards highly expressed genes. a Schematic illustration of the creation of ‘pseudobulks’ from single-cell data. Top, biological replicate from which each cell was obtained. Bottom, simulated gene expression matrix. Read counts for each gene are aggregated across cells of a given type within each biological replicate. b Mean AUCCs across eighteen ground-truth datasets after dividing the transcriptome into terciles of lowly, moderately, or highly expressed genes. c Mean expression levels of the 100 top-ranked false-positive genes from each DE method. d Spearman correlation between the mean expression of 80 ERCC spike-ins expressed in at least three cells and the –log10 p-value of differential expression assigned by each DE method. e Scatterplots of mean ERCC expression vs. –log10 p-value for exemplary single-cell and pseudobulk DE methods. Trend lines and shaded areas show local polynomial regression and the 95% confidence interval, respectively. f Mean expression levels of the 200 top-ranked genes from each DE method in a collection of 46 scRNA-seq datasets.CC BY · PubMed Central
Fig. 3 DE analysis of single-cell data must account for biological replicates. a Schematic illustration of the experiment shown in b, in which the aggregation procedure was disabled and pseudobulk DE methods were applied to individual cells. b Left, AUCC of the original fourteen DE methods, plus six pseudobulk methods applied to individual cells, in the eighteen ground-truth datasets. Right, Spearman correlation between ERCC mean expression and –log10 p-value assigned by six pseudobulk DE methods, before and after disabling the aggregation procedure. c Schematic illustration of the experiment shown in d, in which the replicate associated with each cell was shuffled to produce ‘pseudo-replicates.’ d Left, AUCC of the original fourteen DE methods, plus six pseudobulk methods applied to pseudo-replicates, in the eighteen ground-truth datasets. Right, Spearman correlation between ERCC mean expression and –log10 p-value assigned by six pseudobulk DE methods, before and after shuffling replicates to produce pseudo-replicates. e Variance of gene expression in pseudobulks formed from biological replicates and pseudo-replicates in mouse bone marrow mononuclear cells stimulated with poly-I:C. Shuffling the replicate associated with each cell produced a systematic decrease in the variance of gene expression. Right, pie chart shows the proportion of genes with increased or decreased variance in pseudo-replicates, as compared to biological replicates. f Decreases in the variance of gene expression in pseudo-replicates as compared to biological replicates across 46 scRNA-seq datasets. Points show the mean variance in biological replicates; arrowheads show the mean variance in pseudo-replicates. g Left, expression of the gene c in biological replicates (points) and pseudo-replicates (arrowheads) from unstimulated cells and cells stimulated with poly-I:C, with the range of possible pseudo-replicate expression values shown as a density. Right, mean (horizontal line) and variance (shaded area) of Txnrd3 expression in biological replicates (left) and pseudo-replicates (right). P-values were calculated by edgeR-LRT.
Fig. 3 DE analysis of single-cell data must account for biological replicates. a Schematic illustration of the experiment shown in b, in which the aggregation procedure was disabled and pseudobulk DE methods were applied to individual cells. b Left, AUCC of the original fourteen DE methods, plus six pseudobulk methods applied to individual cells, in the eighteen ground-truth datasets. Right, Spearman correlation between ERCC mean expression and –log10 p-value assigned by six pseudobulk DE methods, before and after disabling the aggregation procedure. c Schematic illustration of the experiment shown in d, in which the replicate associated with each cell was shuffled to produce ‘pseudo-replicates.’ d Left, AUCC of the original fourteen DE methods, plus six pseudobulk methods applied to pseudo-replicates, in the eighteen ground-truth datasets. Right, Spearman correlation between ERCC mean expression and –log10 p-value assigned by six pseudobulk DE methods, before and after shuffling replicates to produce pseudo-replicates. e Variance of gene expression in pseudobulks formed from biological replicates and pseudo-replicates in mouse bone marrow mononuclear cells stimulated with poly-I:C. Shuffling the replicate associated with each cell produced a systematic decrease in the variance of gene expression. Right, pie chart shows the proportion of genes with increased or decreased variance in pseudo-replicates, as compared to biological replicates. f Decreases in the variance of gene expression in pseudo-replicates as compared to biological replicates across 46 scRNA-seq datasets. Points show the mean variance in biological replicates; arrowheads show the mean variance in pseudo-replicates. g Left, expression of the gene c in biological replicates (points) and pseudo-replicates (arrowheads) from unstimulated cells and cells stimulated with poly-I:C, with the range of possible pseudo-replicate expression values shown as a density. Right, mean (horizontal line) and variance (shaded area) of Txnrd3 expression in biological replicates (left) and pseudo-replicates (right). P-values were calculated by edgeR-LRT.CC BY · PubMed Central
Fig. 4 False discoveries in single-cell DE. a Schematic illustration of simulation experiments. Single-cell RNA-seq datasets were simulated with varying degrees of heterogeneity between replicates. Replicates were then randomly assigned to either a ‘treatment’ or ‘control’ group, and DE analysis was performed between groups. b Number of DE genes detected in stimulation experiments with varying degrees of heterogeneity between replicates by a representative single-cell DE method, a representative pseudobulk method, and the same pseudobulk method applied to pseudo-replicates. Points and error bars show the mean and standard deviation across ten independent simulations. c Number of DE genes detected by the tests shown in b for genes divided into deciles by the magnitude of the change in variance between biological replicates and pseudo-replicates (∆-variance). d Volcano plots showing DE between T cells from random groups of unstimulated controls drawn from Kang et al.5 using a representative pseudobulk method, edgeR-LRT, applied to biological replicates or pseudo-replicates. Discarding information about biological replicates leads to the appearance of false discoveries. e Number of DE genes detected in comparisons of random groups of unstimulated controls from 14 scRNA-seq studies with at least six control samples. f Number of DE genes detected within spinal cord regions from control mice profiled by spatial transcriptomics24 using a representative pseudobulk method, edgeR-LRT (points), or a representative single-cell method, the Wilcoxon rank-sum test (arrowheads). g Mean change in variance between biological replicates and pseudo-replicates for 18 human and 20 mouse scRNA-seq datasets.
Fig. 4 False discoveries in single-cell DE. a Schematic illustration of simulation experiments. Single-cell RNA-seq datasets were simulated with varying degrees of heterogeneity between replicates. Replicates were then randomly assigned to either a ‘treatment’ or ‘control’ group, and DE analysis was performed between groups. b Number of DE genes detected in stimulation experiments with varying degrees of heterogeneity between replicates by a representative single-cell DE method, a representative pseudobulk method, and the same pseudobulk method applied to pseudo-replicates. Points and error bars show the mean and standard deviation across ten independent simulations. c Number of DE genes detected by the tests shown in b for genes divided into deciles by the magnitude of the change in variance between biological replicates and pseudo-replicates (∆-variance). d Volcano plots showing DE between T cells from random groups of unstimulated controls drawn from Kang et al.5 using a representative pseudobulk method, edgeR-LRT, applied to biological replicates or pseudo-replicates. Discarding information about biological replicates leads to the appearance of false discoveries. e Number of DE genes detected in comparisons of random groups of unstimulated controls from 14 scRNA-seq studies with at least six control samples. f Number of DE genes detected within spinal cord regions from control mice profiled by spatial transcriptomics24 using a representative pseudobulk method, edgeR-LRT (points), or a representative single-cell method, the Wilcoxon rank-sum test (arrowheads). g Mean change in variance between biological replicates and pseudo-replicates for 18 human and 20 mouse scRNA-seq datasets.CC BY · PubMed Central
Fig. 5 True and false discoveries in the injured mouse spinal cord. a Synapses and projections of Vglut2ON neurons in the mouse lumbar spinal cord before and after SCI. Experiments were repeated in n = 3 mice. b Top, chronophotography of mice before and after SCI. Bottom left, principal component analysis of gait parameters for each condition (small circles). Large circles show the average per group. Bottom right, bar plot showing the average scores on principal component 1 (PC1), which quantify the locomotor performance of injured and uninjured mice. c snRNA-seq experimental design. Inset demonstrates persistent amplitude at increasing frequency of the H-reflex in response to plantar stimulation, reflecting hyperexcitability. d Uniform manifold approximation and projection (UMAP) visualization of 19,237 nuclei, revealing the major cell types of the mouse lumbar spinal cord. e UMAP visualization colored by the AUC of cell type prioritizations, as calculated by Augur27. Endothelial cells are highlighted as the cell type with the highest AUC. f Pseudobulk expression levels in lumbar spinal cord endothelial cells by snRNA-seq (in counts per 10,000), left, and mean counts per Pecam1ON cell by RNA-scope, center, for Igfbp6, a gene identified as DE only by edgeR-LRT. Horizontal line and shaded area show the mean and standard deviation, respectively. Right, colocalization of Igfbp6 with the endothelial marker gene Pecam1 in RNAscope in situ hybridizations from injured and uninjured mouse spinal cords. g As in f, but for Prex2, a gene identified as DE only by the Wilcoxon rank-sum test. h Summary of in vivo screen results. Filled bars represent genes validated by RNAscope. i, Immunohistochemistry reveals Pecam1ON cells with atrophic features after SCI. Experiments were repeated in n = 3 mice. Source data are provided as a Source Data file.
Fig. 5 True and false discoveries in the injured mouse spinal cord. a Synapses and projections of Vglut2ON neurons in the mouse lumbar spinal cord before and after SCI. Experiments were repeated in n = 3 mice. b Top, chronophotography of mice before and after SCI. Bottom left, principal component analysis of gait parameters for each condition (small circles). Large circles show the average per group. Bottom right, bar plot showing the average scores on principal component 1 (PC1), which quantify the locomotor performance of injured and uninjured mice. c snRNA-seq experimental design. Inset demonstrates persistent amplitude at increasing frequency of the H-reflex in response to plantar stimulation, reflecting hyperexcitability. d Uniform manifold approximation and projection (UMAP) visualization of 19,237 nuclei, revealing the major cell types of the mouse lumbar spinal cord. e UMAP visualization colored by the AUC of cell type prioritizations, as calculated by Augur27. Endothelial cells are highlighted as the cell type with the highest AUC. f Pseudobulk expression levels in lumbar spinal cord endothelial cells by snRNA-seq (in counts per 10,000), left, and mean counts per Pecam1ON cell by RNA-scope, center, for Igfbp6, a gene identified as DE only by edgeR-LRT. Horizontal line and shaded area show the mean and standard deviation, respectively. Right, colocalization of Igfbp6 with the endothelial marker gene Pecam1 in RNAscope in situ hybridizations from injured and uninjured mouse spinal cords. g As in f, but for Prex2, a gene identified as DE only by the Wilcoxon rank-sum test. h Summary of in vivo screen results. Filled bars represent genes validated by RNAscope. i, Immunohistochemistry reveals Pecam1ON cells with atrophic features after SCI. Experiments were repeated in n = 3 mice. Source data are provided as a Source Data file.CC BY · PubMed Central

Verification

Claims the paper makes

21 Valid · 9 Not valid · 4 Not checked · 144 Own results and interpretations

133 more claims · 12 with their cited sources checked

Create a free account to see all 178 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 21
  • Probably not a substantial claim. 21
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

6 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 104 references

Research works printed but not linked: 1

  1. 10.1007/978-1-4939-3578-9$\_$19 · doi:10.1007/978-1-4939-3578-9$_$19

Cited by

See all 1,210

Metadata

Authors
Squair, J. W., Gautier, M., Kathe, C., Anderson, M. A., James, N. D., Hutson, T. H., Hudelle, R., Qaiser, T., Matson, K. J. E., Barraud, Q., Levine, A. J., La Manno, G., Skinnider, M. A., & Courtine, G.
Affiliations
  • University of British Columbia (CA)
  • Centre Hospitalier Universitaire Vaudois (CH)
  • National Institute of Neurological Disorders and Stroke (US)
Corresponding author
  • Skinnider, Michael A.
  • Courtine, Grégoire
Venue
Nature Communications
Publisher
openRxiv
Published
2021-03-12
Volume
12
Issue
1
Pages
5692-5692
ISSN
2041-1723 (ISSN-L)
OpenAlex
W3202649987
Cited by
1,086
References
104
Citations by year
2026: 396 · 2025: 214 · 2024: 231 · 2023: 145 · 2022: 86 · 2021: 12
FWCI
34.62
Type
journal article
Language
en
Open access
Locations
  • Nature Communications (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (submittedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • SERVAL (Université de Lausanne) (publishedVersion, cc-by)
  • IRIS (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
Funders
  • National Science Foundation
  • Wings for Life
  • Compute Canada
  • Western Canada Research Grid
  • Wyss Center for Bio and Neuroengineering
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (185728, 192558, 193445, 190495, 310030)
  • University of British Columbia
  • National Institutes of Health
  • Canadian Institutes of Health Research
  • National Supercomputing Center, Korea Institute of Science and Technology Information
  • National Institute of Neurological Disorders and Stroke
  • H2020 European Research Council (682999)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
Keywords
single-cell differential expression, false discovery, biological replicates, comparison of statistical methods, differentially expressed genes, spinal cord injury, mouse spinal cord, cell type-specific responses, false positives
MeSH terms
Data Accuracy, Biological Variation, Population, Biological Variation, Individual, RNA-Seq, Animals, Gene Expression Regulation, Humans, Rabbits, Swine, Models, Statistical, Mice, Rats, Single-Cell Analysis, Datasets as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Confronting false discoveries in single-cell differential expression. https://app.cochat.ai/papers/10.1101/2021.03.12.435024
The paper
Squair, J. W., Gautier, M., Kathe, C., Anderson, M. A., James, N. D., Hutson, T. H., Hudelle, R., Qaiser, T., Matson, K. J. E., Barraud, Q., Levine, A. J., La Manno, G., Skinnider, M. A., & Courtine, G. (2021). Confronting false discoveries in single-cell differential expression. Nature Communications. https://doi.org/10.1101/2021.03.12.435024

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.