Confronting false discoveries in single-cell differential expression

Within the compendium, fourteen studies had control groups that included six or more samples.

“Within our compendium, we identified a total of fourteen studies with control groups that included six or more samples5,6,15,49–59.”

• cites [5] sourceSupported
• cites [6] 2019, Single-cell transcriptomic analysis of Alzheimer’s disease, Nature sourceSupported
• cites [15] sourceSupported
• cites [49] 2020, Single-nucleus transcriptomics of the prefrontal cortex in major depressive disorder implicates oligodendrocyte precursor cells and excitatory neurons, Nat. Neu-rosci sourceSupported
• cites [50] sourceSupported
• cites [51] sourcePartly supported

  The Fig. 9 caption reports six Sham/Veh controls for behavioral tests, so this paper does contain a control group of six. But the Methods report only three Sham samples in the Drop-seq data and four per group selected for T4-treatment sequencing. The paper cannot establish the citing authors’ count of fourteen studies or show that this study had six control sequencing samples. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [52] 2018, Single-cell RNA-seq uncovers a robust transcriptional response to morphine by glia, Cell Rep sourceDisputed

  The cited paper is one study, so it cannot establish the compendium-wide total of fourteen. For its own control group, STAR⋆METHODS reports four mock-treatment biological replicates in the single-cell analysis, not six or more samples; its bulk RNA-seq analysis reports three mock-treatment replicates. (LLM, conf=0.86)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured
• cites [53] sourcePartly supported

  The study description (section labeled Abstract; page not provided) reports uninjured, intact-tail samples alongside amputated-tail samples, supporting its identification as a study with a control condition. It says there were at least two biological replicates per condition, but does not establish that the control group had six or more samples. The count of fourteen studies is the citing authors’ compendium count, not a finding attributed to this individual source. (LLM, conf=0.77)

  Experimental — accuracy not yet measured
• cites [54] 2019, Cell type-specific transcriptional programs in mouse prefrontal cortex during adolescence and addiction, Nat. Commun sourceSupported
• cites [55] 2020, Single cell transcriptome profiling of the human alcohol-dependent brain, Hum. Mol. Genet sourceDisputed

  The Results and Discussion section (page unavailable) says this study included four control donors, not six or more. The Conclusions section likewise describes seven donors in total: three with alcohol dependence and four controls. This source therefore cannot support its inclusion as a study meeting the stated control-group threshold; it does not address the compendium-wide count of fourteen. (LLM, conf=0.91)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: The 46 published scRNA-seq datasets in the compendium are representative of single-cell RNA-seq studies generally.

Pseudobulk methods more accurately predicted changes in protein abundance than single-cell methods.

“Moreover, comparisons to matching proteomics data 13 revealed that pseudobulk methods also more accurately predicted changes in protein abundance (Supplementary Fig. 1e-f ).”

• cites [13] 2018, Gene expression variability across cells and species shapes innate immunity, Nature sourceSupported

Within the same experimental condition, replicates exhibit inherent differences in gene expression, which reflect both biological and technical factors.

“Within the same experimental condition, replicates exhibit inherent differences in gene expression, which reflect both biological and technical factors 22 .”

• cites [22] sourceSupported

All six of the top-performing DE methods aggregated cells within a biological replicate to form pseudobulks before applying a statistical test.

“This analysis immediately revealed that all six of the top-performing methods shared a common analytical property. These methods aggregated cells within a biological replicate, to form so-called ‘pseudobulks’, before applying a statistical test (Fig. 1c ) 18 .”

• cites [18] sourceSupported

Single-cell methods failed to identify the relevant GO term when comparing mouse phagocytes stimulated with poly(I:C).

“For example, single-cell methods failed to identify the relevant GO term when comparing mouse phagocytes stimulated with poly(I:C) 12 , a synthetic double-stranded RNA (Fig. 1f ).”

• cites [12] sourceCould not verify

  The Methods section on bone marrow-derived phagocytes reports poly(I:C) stimulation of rodent cells and single-cell analysis, but does not report whether a GO term was identified. Searches for GO-term enrichment and a failure to identify a term found no passage addressing the claimed result. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

An extensive survey of the literature identified a total of eighteen 'gold standard' datasets that met the criteria of matched bulk and scRNA-seq performed on the same population of purified cells, exposed to the same perturbations, and sequenced in the same laboratories.

“An extensive survey of the literature identified a total of eighteen ‘gold standard’ datasets that met these criteria (Fig. 1a ) 12 – 15 .”

• cites [12] sourceSupported
• cites [13] 2018, Gene expression variability across cells and species shapes innate immunity, Nature sourceSupported
• cites [14] 2020, Single-cell transcriptomics identifies an effectorness gradient shaping the response of CD4+ T cells to cytokines, Nat. Commun sourceSupported
• cites [15] sourcePartly supported

  Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

Cell type prioritization using Augur indicated that endothelial cells underwent the most profound transcriptional changes in the spinal cord below the injury.

“This unbiased analysis indicated that endothelial cells underwent the most profound transcriptional changes in the spinal cord below the injury (Fig. 5e ).”

• cites [27] sourcePartly supported

  The paper’s Abstract describes applying Augur to lumbar spinal-cord cells after injury, but its spinal-cord prioritization focuses on interneuron subtypes. The Abstract reports profound changes in endothelial cells in Alzheimer’s disease datasets, not in the spinal cord below an injury. It therefore supports the use of Augur in a spinal-cord injury setting, but not the attributed endothelial-cell finding. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [28] 2020, A curated database reveals trends in single-cell transcriptomics, Database sourceNot checked — no full text of the source

Several validated edgeR-LRT genes, including Slc7a11 and Igfbp6, are involved in the response to hypoxia within endothelial cells.

“Several of the validated edgeR-LRT genes, including Slc7a11 and Igfbp6 , are involved in the response to hypoxia within endothelial cells, supporting the establishment of a chronically hypoxic state in the lumbar spinal cord 31 – 33 .”

• cites [31] sourcePartly supported

  The section “HIF regulates antioxidant defenses in hypoxic cells” (page unavailable) says hypoxia induces SLC7A11 expression in breast cancer cells. It does not establish that SLC7A11 or Igfbp6 responds to hypoxia in endothelial cells, and the retrieved passages do not address Igfbp6. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [32] sourceSupported
• cites [33] sourceCould not verify

  The Results section, “Endogenous trace amines induce hypoxia after SCI” (page unavailable), reports chronic hypoxia in spinal cord tissue caudal to injury. But the paper’s mRNA analysis (Online Methods; page unavailable) does not identify Slc7a11 or Igfbp6 or describe either gene’s role in endothelial-cell responses to hypoxia. Searches for both gene names found no matching passage. (LLM, conf=0.75)

  Experimental — accuracy not yet measured

Previous studies on this issue relied almost entirely on synthetic data, supplemented by a few illustrative case studies.

“However, these studies relied almost entirely on synthetic data, supplemented by a few illustrative case studies.”

• cites [18] sourceSupported
• cites [34] sourcePartly supported

  The Results (Simulation; no page available) describe this paper’s simulation study, and the Discussion (no page available) says it also examined a few independent real datasets. That supports the characterization of this paper’s approach, but the copy does not establish that previous studies generally relied almost entirely on synthetic data or used real data only as illustrative case studies. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

The most widely used analysis packages in the field currently employ DE methods prone to false discoveries by default.

“the most widely used analysis packages in the field currently employ DE methods prone to false discoveries by default 35 , 36 .”

• cites [35] sourceCould not verify

  The STAR Methods (Quantification and statistical analysis; page unavailable) describe differential-expression testing with Seurat’s FindMarkers, but specify a test rather than identifying its default. Our copy does not assess whether widely used packages’ default DE methods are prone to false discoveries (searched: false discoveries, false positives, and default DE tests). (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [36] sourcePartly supported

  The Results section ('SCANPY integrates canonical analysis methods in a scalable way'; page unavailable) says SCANPY identifies marker genes using differential expression tests. The Background also identifies established single-cell analysis frameworks. Neither passage says the most widely used packages employ such tests by default or that their methods are prone to false discoveries. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Changes in mRNA abundance within multicellular tissues are confounded by different responses across cell types and changes in the relative abundance of these cell types.

“Changes in mRNA abundance within multicellular tissues are confounded by different responses across cell types and changes in the relative abundance of these cell types3.”

• cites [3] sourceSupported

Differences in the specification of the simulation model have led investigators to contrasting conclusions about differential expression methods.

“Differences in the specification of this model have led investigators to contrasting conclusions9,10.”

• cites [9] 2020, muscat detects subpopulation-specific state transitions from multi-sample multi-condition singlecell transcriptomics data, Nat. Commun sourceCould not verify

  The Introduction and Results (Simulation framework; pages unavailable) describe a flexible model used to compare methods across simulation scenarios. The Discussion (page unavailable) notes that other scenarios could be tested, but the passages found do not say that differences in model specification led investigators to contrasting conclusions. I searched for differing simulation assumptions and conflicting method comparisons. (LLM, conf=0.70)

  Experimental — accuracy not yet measured
• cites [10] 2021, A practical solution to pseudoreplication bias in single-cell studies, Nat. Commun sourceCould not verify

  The Introduction describes this paper’s simulation of differential-expression methods, and the Methods describe its simulation model. Neither passage says that differences between investigators’ simulation-model specifications led them to contrasting conclusions. Searches for conflicting findings across simulation studies found no passage addressing that claim. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Pseudobulk methods more accurately predicted changes in protein abundance than single-cell methods in comparisons to matching proteomics data.

“Moreover, comparisons to matching proteomics data13 revealed that pseudobulk methods also more accurately predicted changes in protein abundance (Supplementary Fig. 1e-f).”

• cites [13] 2018, Gene expression variability across cells and species shapes innate immunity, Nature sourceCould not verify

  The Methods section on bone marrow-derived phagocytes says differential expression was analyzed using an in silico bulk made from single-cell data. Neither that passage nor passages retrieved by searching for proteomics and protein abundance report a comparison with matching proteomics data or show that pseudobulk methods predicted protein changes more accurately. (LLM, conf=0.74)

  Experimental — accuracy not yet measured

An extensive literature survey identified eighteen gold standard datasets that met the criteria of matched bulk and scRNA-seq on the same purified cell population, exposed to the same perturbations, and sequenced in the same laboratories.

“An extensive survey of the literature identified a total of eighteen ‘gold standard’ datasets that met these criteria (Fig. 1a)12–15.”

• cites [12] sourceCould not verify

  The Methods section describes a catalog of gene sets, and the Results section describes six GSEA examples; neither reports the claimed literature survey. Searches for eighteen matched bulk and scRNA-seq datasets and their stated selection criteria found no supporting passage in our copy. (LLM, conf=0.77)

  Experimental — accuracy not yet measured
• cites [13] 2018, Gene expression variability across cells and species shapes innate immunity, Nature sourceSupported
• cites [14] 2020, Single-cell transcriptomics identifies an effectorness gradient shaping the response of CD4+ T cells to cytokines, Nat. Commun sourceSupported
• cites [15] sourceSupported

• AssumptionThe authors state this as an assumption: The closest possible approximation to the experimental ground truth can be obtained from matched bulk and scRNA-seq performed on the same population of purified cells, exposed to the same perturbations, and sequenced in the same laboratories.

Failing to account for the intrinsic variability of biological replicates produced hundreds of DE genes between randomly assigned replicates in control human PBMC samples not exposed to interferon.

“Failing to account for the intrinsic variability of biological replicates produced hundreds of DE genes between randomly assigned replicates (Fig. 4d and Supplementary Fig. 6a, b).”

• cites [5] sourcePartly supported

  The paper reports substantial inter-individual expression variability in control PBMCs, including 15–827 genes with statistically significant variability (results passage, labeled Abstract in our copy; page unavailable). But those are genes variable across individuals, not hundreds of differentially expressed genes found between randomly assigned control replicates. The Methods describe using individuals as replicates for differential-expression analysis, but do not report the claimed random-assignment comparison. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Single-cell methods incorrectly identified many abundant spike-ins as DE in a dataset with equal concentrations of synthetic mRNAs spiked into each single-cell library.

“We found that single-cell methods incorrectly identified many abundant spike-ins as DE (Fig. 2d-e and Supplementary Fig. 2e-f).”

• cites [12] sourceCould not verify

  The Abstract discusses differential expression in microarray profiles, and the Methods describe gene-set enrichment analysis. A search of our copy for single-cell methods, synthetic mRNA spike-ins, and false differential-expression calls found no passage addressing the claimed result. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [21] sourcePartly supported

  The Differential expression analysis section (page unavailable) uses synthetic spike-ins at a 1:1 mixing ratio as true negatives, and the Null model evaluation section reports false-positive DE calls in technical replicates. The paper does not report that single-cell methods called many abundant, equally concentrated spike-ins DE; its reported false positives come from a different comparison. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Previous studies on conflating variability between replicates with biological effect relied almost entirely on synthetic data, supplemented by a few illustrative case studies.

“However, these studies relied almost entirely on synthetic data, supplemented by a few illustrative case studies.”

• cites [18] sourceNot checked — no full text of the source
• cites [34] sourceCould not verify

  The Introduction and Results > Statistical principles discuss biological replicates, variability, and methods for analysing gene expression, but do not characterize the evidence used by previous studies. A search for synthetic or simulated data and illustrative case studies found no passage saying those studies relied almost entirely on synthetic data. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

The emergence of a bias towards highly expressed genes paralleled this decrease in performance.

“The emergence of a bias towards highly expressed genes paralleled this decrease in performance (Fig. 3b and Supplementary Fig. 4b-c ).”

The genes falsely identified as DE corresponded to those with the highest variability between replicates.

“Moreover, we again confirmed that the genes falsely identified as DE corresponded to those with the highest variability between replicates (Supplementary Fig. 6d ).”

The resulting DE genes were enriched for hundreds of Gene Ontology (GO) terms despite a complete absence of biological perturbation.

“The resulting DE genes were enriched for hundreds of Gene Ontology (GO) terms, despite a complete absence of biological perturbation (Supplementary Fig. 6c ).”

In a spike-in dataset, single-cell methods incorrectly identified many abundant spike-ins as DE.

“We found that single-cell methods incorrectly identified many abundant spike-ins as DE (Fig. 2d-e and Supplementary Fig. 2e-f ).”

Pseudobulk methods dramatically reduce the number of zeros in the data, especially for lowly expressed genes.

“By aggregating cells within each replicate, pseudobulk methods dramatically reduce the number of zeros in the data, especially for lowly expressed genes (Fig. 2a ).”

Pseudobulk methods abolished these false discoveries.

“In contrast, pseudobulk methods abolished these false discoveries.”

Pseudobulk methods abolished the false detection of DE genes.

“Pseudobulk methods abolished the false detection of DE genes.”

Shuffling the replicates produced a systematic decrease in the variance of gene expression, affecting 98.2% of genes in a dataset of bone marrow mononuclear cells stimulated with poly-I:C.

“We found that shuffling the replicates produced a systematic decrease in the variance of gene expression, affecting 98.2% of genes (Fig. 3e ).”

In a dataset of human PBMCs exposed to interferon, failing to account for the intrinsic variability of biological replicates produced hundreds of DE genes between randomly assigned replicates.

“Failing to account for the intrinsic variability of biological replicates produced hundreds of DE genes between randomly assigned replicates (Fig. 4d and Supplementary Fig. 6a, b ).”

Single-cell DE methods displayed a systematic preference for highly expressed genes across the entire compendium of 46 scRNA-seq datasets.

“We found that single-cell DE methods displayed a systematic preference for highly expressed genes across the entire compendium (Fig. 2f ).”

Pseudobulk methods required only minutes per cell type in the compendium of 46 datasets.

“In contrast, pseudobulk methods required only minutes per cell type in our compendium of 46 datasets (Supplementary Fig. 10e-f ).”

Disabling the aggregation procedure and applying pseudobulk methods to individual cells abolished the superiority of the pseudobulk methods.

“Strikingly, this procedure abolished the superiority of the pseudobulk methods (Fig. 3b and Supplementary Fig. 4a ).”

In simulated single-cell data with different degrees of heterogeneity between replicates in the absence of any difference between groups, single-cell methods identified hundreds of DE genes in the absence of any perturbation.

“Strikingly, single-cell methods identified hundreds of DE genes in the absence of any perturbation (Fig. 4b and Supplementary Fig. 4a ).”

They detected significantly more variability between replicates in human datasets than in mouse datasets across 41 human and mouse scRNA-seq datasets.

“In agreement with our hypothesis, we detected significantly more variability between replicates in the human datasets (Fig. 4g ).”

In datasets comprising 500 or more cells, GLMM performance converged to that of pseudobulk DE methods.

“However, in datasets comprising 500 or more cells, their performance converged to that of pseudobulk DE methods.”

The Wilcoxon rank-sum test and edgeR-LRT identified largely distinct sets of DE genes, with only four genes overlapping between the two methods.

“These methods identified largely distinct sets of DE genes, with only four genes overlapping between the two methods.”

Methods that compared individual cells performed poorly.

“In comparison, methods that compared individual cells performed poorly.”

Pseudobulk methods more faithfully reflected the ground truth in GO term enrichment analyses, as captured in bulk RNA-seq.

“We found that pseudobulk methods again more faithfully reflected the ground truth, as captured in the bulk RNA-seq (Fig. 1e and Supplementary Fig. 1g ).”

They detected the presence of numerous atrophic blood vessels below the level of injury.

“In line with the expected consequences of chronic hypoxia, we detected the presence of numerous atrophic blood vessels below the level of injury (Fig. 5i ).”

Highly expressed genes exhibited the largest decrease in variance in pseudo-replicates.

“Moreover, we found that highly expressed genes exhibited the largest decrease in variance in pseudo-replicates, thus explaining the bias of single-cell methods towards highly expressed genes (Supplementary Fig. 4g-k ).”

Multifactorial quantification of whole-body kinematics revealed profound impairments in the ability of mice to produce locomotion after severe contusion of the mid-thoracic spinal cord.

“Multifactorial quantification of whole-body kinematics revealed profound impairments in the ability of the mice to produce locomotion (Fig. 5b and Supplementary Fig. 8a ).”

The most pronounced differences between pseudobulk and single-cell methods emerged among highly expressed genes.

“Instead, the most pronounced differences between pseudobulk and single-cell methods emerged among highly expressed genes.”

Only three of thirteen genes called as DE by the Wilcoxon rank-sum test could be corroborated (p < 0.05, χ2 test).

“In marked contrast, only three of thirteen genes called as DE by the Wilcoxon rank-sum test could be corroborated (p < 0.05, χ 2 test; Fig. 5f-h ).”

In a spatial transcriptomics dataset of ALS spinal cords, statistical methods that failed to account for variability between biological replicates identified thousands of DE genes within each region.

“Statistical methods that failed to account for variability between biological replicates identified thousands of DE genes within each region (Fig. 4f and Supplementary Fig. 6e ).”

In the poly-I:C dataset, failing to account for the variable expression of Txnrd3 across replicates led to the spurious identification of this gene as differentially expressed.

“For instance, in the poly-I:C dataset, failing to account for the variable expression of Txnrd3 across replicates led to the spurious identification of this gene as differentially expressed (Fig. 3g ).”

The number of false discoveries was reduced when additional replicates were introduced to the dataset.

“The number of false discoveries was reduced when additional replicates were introduced to the dataset (Supplementary Fig. 4c ).”

They sequenced a total of 19,237 cells that encompassed all the major cell types of the lumbar spinal cord.

“We sequenced a total of 19,237 cells that encompassed all the major cell types of the lumbar spinal cord (Fig. 5d ).”

False positives identified by single-cell DE methods were more highly expressed than those identified by pseudobulk methods.

“We found that false positives identified by single-cell DE methods were more highly expressed than those identified by pseudobulk methods (Fig. 2c and Supplementary Fig. 2b ).”

In datasets of 25-50 cells, GLMMs could produce accurate results under very specific parameter combinations.

“In datasets of 25-50 cells, GLMMs could produce accurate results under very specific parameter combinations.”

RNAscope validated five of the six genes called as DE by edgeR-LRT.

“RNAscope validated five of the six genes called as DE by edgeR-LRT.”

Creating pseudo-replicates led to the reappearance of spurious DE genes.

“However, creating pseudo-replicates led to the reappearance of spurious DE genes (Fig. 4b-c and Supplementary Fig. 4a-b ), further corroborating the requirements for accurate DE analyses.”

False-negatives overlooked by single-cell DE methods tended to be lowly expressed.

“Conversely, false-negatives overlooked by single-cell DE methods tended to be lowly expressed (Supplementary Fig. 2c-d ).”

Every comparison across the compendium of 46 datasets displayed the same decrease in the variance of gene expression.

“Every comparison displayed the same decrease in the variance of gene expression (Fig. 3f ).”

There were minimal differences between pseudobulk and single-cell methods for lowly expressed genes.

“Contrary to our prediction, we observed minimal differences between pseudobulk and single-cell methods for lowly expressed genes (Fig. 2b and Supplementary Fig. 2a ).”

Pseudobulk methods avoided this bias.

“In contrast, pseudobulk methods avoided this bias.”

Introducing additional cells to the simulated data only exacerbated the underlying problem.

“In contrast, introducing additional cells to the simulated data only exacerbated the underlying problem (Supplementary Fig. 4d ).”

The genes falsely called as DE were those whose expression was most variable between replicates.

“Moreover, in line with our understanding of the mechanisms underlying the failure of single-cell DE methods, the genes that were falsely called as DE were those whose expression was most variable between replicates (Fig. 4c and Supplementary Fig. 4b ).”

The variance of the most variable genes is disproportionately underestimated when discarding information about biological replicates.

“We performed a similar analysis correlating the original variance of gene expression to the ∆-variance, demonstrating that the variance of the most variable genes is disproportionately underestimated when discarding information about biological replicates.”

• AssumptionRests on an assumption: The 46 published scRNA-seq datasets in the compendium are representative of single-cell RNA-seq studies generally.

Some statistical tests returned a large number of p-values below the double precision limit in R (approximately 2 × 10-308).

“We observed that some statistical tests returned a large number of p-values below the double precision limit in R (approximately 2 × 10 –308 )”

Most studies published in the past two years have used inappropriate statistical methods for DE analysis.

“our observation that most studies published in the past two years have used inappropriate statistical methods for DE analysis.”

• AssumptionRests on an assumption: The 500 most recent entries in the curated database are representative of scRNA-seq studies published in the past two years.

Pseudobulk methods achieve the highest fidelity to the experimental ground truth at the levels of the transcriptome, proteome, and functional interpretation.

“we found that pseudobulk methods achieve the highest fidelity to the experimental ground truth at the levels of the transcriptome, proteome, and functional interpretation.”

• AssumptionRests on an assumption: The 18 ground truth datasets are representative of single-cell experiments for assessing DE methods.

The mean ∆-variance across all genes decreased in all 46 datasets in the scRNA-seq compendium.

“we calculated the mean ∆-variance across all genes in each of the 46 datasets in our scRNA-seq compendium, observing a decrease in the variance in all 46 cases.”

The performance of DE methods is contingent on their ability to account for biological replicates.

“Having established that the performance of DE methods is contingent on their ability to account for biological replicates, we asked why mixed models failed to match the performance of pseudobulk methods.”

• AssumptionRests on an assumption: The 18 ground truth datasets are representative of single-cell experiments for assessing DE methods.
• AssumptionRests on an assumption: The 46 published scRNA-seq datasets in the compendium are representative of single-cell RNA-seq studies generally.

Most DE genes identified by the most commonly used statistical method are false discoveries in a systematic in vivo screen of the injured mouse spinal cord.

“we show through a systematic in vivo screen of the injured mouse spinal cord that most DE genes identified by the most commonly used statistical method are false discoveries.”

The variance of highly expressed genes is disproportionately underestimated when discarding information about biological replicates.

“We observed a significant negative correlation, confirming that the variance of highly expressed genes is disproportionately underestimated when discarding information about biological replicates.”

• AssumptionRests on an assumption: The 46 published scRNA-seq datasets in the compendium are representative of single-cell RNA-seq studies generally.

Methods that compared individual cells performed poorly compared to pseudobulk methods.

“In comparison, methods that compared individual cells performed poorly.”

False negatives overlooked by single-cell DE methods tended to be lowly expressed.

“Conversely, false-negatives overlooked by single-cell DE methods tended to be lowly expressed (Supplementary Fig. 2c-d).”

The six top-performing DE methods aggregated cells within a biological replicate to form pseudobulks before applying a statistical test.

“These methods aggregated cells within a biological replicate, to form so-called ‘pseudobulks’, before applying a statistical test (Fig. 1c)18.”

• cites [18] sourceNot checked — no full text of the source

The fourteen selected DE methods have been used by almost 90% of recent scRNA-seq studies.

“Together, these methods have been used by almost 90% of recent studies (Fig. 1b).”

The differences between pseudobulk and single-cell DE methods were highly significant.

“The differences between pseudobulk and single-cell methods were highly significant (Fig. 1d), and robust to the methodology used to quantify concordance (Supplementary Fig. 1a-d).”

The most widely used single-cell differential expression methods can discover hundreds of differentially expressed genes in the absence of biological differences.

“Indeed, the most widely used methods can discover hundreds of differentially expressed genes in the absence of biological differences.”

Pseudobulk methods more faithfully reflected the ground truth in Gene Ontology term enrichment analyses than single-cell methods.

“We found that pseudobulk methods again more faithfully reflected the ground truth, as captured in the bulk RNA-seq (Fig. 1e and Supplementary Fig. 1g).”

The differences between pseudobulk and single-cell DE methods were robust to the methodology used to quantify concordance.

“The differences between pseudobulk and single-cell methods were highly significant (Fig. 1d), and robust to the methodology used to quantify concordance (Supplementary Fig. 1a-d).”

All six of the top-performing DE methods shared a common analytical property.

“This analysis immediately revealed that all six of the top-performing methods shared a common analytical property.”

Pseudobulk methods dramatically reduce the number of zeros in single-cell data, especially for lowly expressed genes.

“By aggregating cells within each replicate, pseudobulk methods dramatically reduce the number of zeros in the data, especially for lowly expressed genes (Fig. 2a).”

• AssumptionRests on an assumption: Measurements of gene expression in single cells are inherently sparse.

Single-cell methods produced a systematic excess of false positives compared to pseudobulk methods in fourteen datasets with at least six control replicates split randomly into synthetic groups.

“This systematic analysis confirmed that single-cell methods produced a systematic excess of false positives compared to pseudobulk methods (Fig. 4e).”

Applying the aggregation procedure to random groups of cells to form pseudo-replicates re-emerged a bias towards highly expressed genes.

“This experiment induced a similar decrease in the performance of pseudobulk methods, combined with the re-emergence of a bias towards highly expressed genes (Fig. 3d and Supplementary Fig. 4d–f).”

The emergence of a bias towards highly expressed genes paralleled the decrease in performance when aggregation was disabled.

“The emergence of a bias towards highly expressed genes paralleled this decrease in performance (Fig. 3b and Supplementary Fig. 4b-c).”

Single-cell DE methods displayed a systematic preference for highly expressed genes across a compendium of 46 scRNA-seq datasets.

“We found that single-cell DE methods displayed a systematic preference for highly expressed genes across the entire compendium (Fig. 2f).”

The injury triggered the aberrant growth of new synapses throughout lumbar segments.

“We found that the injury triggered the aberrant growth of new synapses throughout lumbar segments, combined with the emergence of abnormal segmental reflexes (Fig. 5a and Supplementary Fig. 8a).”

Applying the aggregation procedure to random groups of cells to form pseudo-replicates induced a similar decrease in the performance of pseudobulk methods.

“This experiment induced a similar decrease in the performance of pseudobulk methods, combined with the re-emergence of a bias towards highly expressed genes (Fig. 3d and Supplementary Fig. 4d–f).”

Every comparison across a compendium of 46 datasets displayed the same decrease in the variance of gene expression when replicates were shuffled.

“Every comparison displayed the same decrease in the variance of gene expression (Fig. 3f).”

The resulting DE genes from unperturbed samples were enriched for hundreds of Gene Ontology terms despite a complete absence of biological perturbation.

“The resulting DE genes were enriched for hundreds of Gene Ontology (GO) terms, despite a complete absence of biological perturbation (Supplementary Fig. 6c).”

Pseudobulk methods abolished the false detection of DE genes in simulated data.

“Pseudobulk methods abolished the false detection of DE genes.”

The genes falsely called as DE by single-cell methods were those whose expression was most variable between replicates.

“Moreover, in line with our understanding of the mechanisms underlying the failure of single-cell DE methods, the genes that were falsely called as DE were those whose expression was most variable between replicates (Fig. 4c and Supplementary Fig. 4b).”

The injury triggered the emergence of abnormal segmental reflexes.

“We found that the injury triggered the aberrant growth of new synapses throughout lumbar segments, combined with the emergence of abnormal segmental reflexes (Fig. 5a and Supplementary Fig. 8a).”

Mice that received a severe contusion of the mid-thoracic spinal cord showed profound impairments in the ability to produce locomotion.

“Multifactorial quantification of whole-body kinematics revealed profound impairments in the ability of the mice to produce locomotion (Fig. 5b and Supplementary Fig. 8a).”

Introducing additional cells to the simulated data exacerbated the underlying problem of false discoveries.

“In contrast, introducing additional cells to the simulated data only exacerbated the underlying problem (Supplementary Fig. 4d).”

The number of false discoveries was reduced when additional replicates were introduced to the simulated data.

“The number of false discoveries was reduced when additional replicates were introduced to the dataset (Supplementary Fig. 4c).”

Pseudobulk methods abolished these false discoveries in the spatial transcriptomics dataset.

“In contrast, pseudobulk methods abolished these false discoveries.”

Significantly more variability between replicates was detected in human datasets than in mouse datasets across 41 human and mouse scRNA-seq datasets.

“In agreement with our hypothesis, we detected significantly more variability between replicates in the human datasets (Fig. 4g).”

• AssumptionRests on an assumption: Many factors that produce variability between biological replicates can be minimized in animal models, including genetic background, environment, intensity and timing of the biological perturbation, and sample processing.
• AssumptionRests on an assumption: Sources of variation between biological replicates are inherently more difficult to control in experiments involving human subjects.

Statistical methods that failed to account for variability between biological replicates identified thousands of DE genes within each region of a spatial transcriptomics dataset from control mice.

“Statistical methods that failed to account for variability between biological replicates identified thousands of DE genes within each region (Fig. 4f and Supplementary Fig. 6e).”

• cites [24] sourceNot checked — no full text of the source

Pseudobulk methods avoided the bias towards identifying abundant spike-ins as DE.

“In contrast, pseudobulk methods avoided this bias.”

Pseudobulk methods required only minutes per cell type in a compendium of 46 datasets.

“In contrast, pseudobulk methods required only minutes per cell type in our compendium of 46 datasets (Supplementary Fig. 10e-f).”

In an in vivo screen of the injured mouse spinal cord, most DE genes identified by the most commonly used statistical method are false discoveries.

“For example, we show through a systematic in vivo screen of the injured mouse spinal cord that most DE genes identified by the most commonly used statistical method are false discoveries.”

Endothelial cells underwent the most profound transcriptional changes in the spinal cord below the injury.

“This unbiased analysis indicated that endothelial cells underwent the most profound transcriptional changes in the spinal cord below the injury (Fig. 5e).”

The computational resources required to fit the best-performing GLMMs were enormous.

“Moreover, the computational resources required to fit the best-performing GLMMs were enormous.”

In a literature review of the 500 most recent scRNA-seq studies, 205 of the 500 studies conducted DE analysis between biological conditions.

“In total, 205 of the 500 studies conducted DE analysis between biological conditions.”

A total of 19,237 cells sequenced encompassed all the major cell types of the lumbar spinal cord.

“We sequenced a total of 19,237 cells that encompassed all the major cell types of the lumbar spinal cord (Fig. 5d).”

Even in downsampled datasets, DE analysis of a single cell type took an average of 13.5 h.

“Even in downsampled datasets, DE analysis of a single cell type took an average of 13.5 h (Supplementary Fig. 10c-d).”

Only three of thirteen genes called as DE by the Wilcoxon rank-sum test could be corroborated by RNAscope.

“In marked contrast, only three of thirteen genes called as DE by the Wilcoxon rank-sum test could be corroborated (p < 0.05, χ2 test; Fig. 5f-h).”

These mechanisms are universal across an additional 46 single-cell RNA-seq studies.

“We demonstrate the universality of these mechanisms in multifaceted datasets from an additional 46 single-cell RNA-seq studies.”

• AssumptionRests on an assumption: The 46 published scRNA-seq datasets in the compendium are representative of single-cell RNA-seq studies generally.

Across all 46 datasets in the scRNA-seq compendium, the mean ∆-variance across all genes showed a decrease in variance.

“Subsequently, we calculated the mean ∆-variance across all genes in each of the 46 datasets in our scRNA-seq compendium, observing a decrease in the variance in all 46 cases.”

• AssumptionRests on an assumption: The 46 published scRNA-seq datasets in the compendium are representative of single-cell RNA-seq studies generally.

Methods that fail to account for the variability of biological replicates can produce hundreds of false discoveries in the absence of any biological difference.

“Conversely, methods that fail to account for the variability of biological replicates can produce hundreds of false discoveries in the absence of any biological difference.”

Both experiments entailed a loss of information about biological replicates.

“We recognized that both experiments entailed a loss of information about biological replicates.”

The central principle underlying valid DE analysis is the ability of statistical methods to account for the intrinsic variability of biological replicates.

“Here, we demonstrate that the central principle underlying valid DE analysis is the ability of statistical methods to account for the intrinsic variability of biological replicates.”

By performing a statistical inference at the level of individual cells, single-cell DE methods conflate variability between biological replicates with the effect of a biological perturbation.

“Our results demonstrate that by performing a statistical inference at the level of individual cells, single-cell DE methods conflate variability between biological replicates with the effect of a biological perturbation.”

Single-cell methods have a systematic tendency to identify highly expressed genes as DE even when their expression remained unchanged.

“Together, these findings implied a systematic tendency for single-cell methods to identify highly expressed genes as DE, even when their expression remained unchanged.”

The presence of these genes supports the establishment of a chronically hypoxic state in the lumbar spinal cord.

“Several of the validated edgeR-LRT genes, including Slc7a11 and Igfbp6 , are involved in the response to hypoxia within endothelial cells, supporting the establishment of a chronically hypoxic state in the lumbar spinal cord 31 – 33 .”

The bias of single-cell methods towards highly expressed genes is explained by the largest decrease in variance in pseudo-replicates for highly expressed genes.

“Moreover, we found that highly expressed genes exhibited the largest decrease in variance in pseudo-replicates, thus explaining the bias of single-cell methods towards highly expressed genes (Supplementary Fig. 4g-k ).”

These observations illustrate the potential for single-cell DE methods to produce false discoveries.

“Together, these observations illustrate the potential for single-cell DE methods to produce false discoveries.”

Failing to account for differences between replicates could lead methods to misattribute the inherent variability between replicates to the effect of the perturbation.

“We reasoned that failing to account for these differences could lead methods to misattribute the inherent variability between replicates to the effect of the perturbation.”

Accounting for biological replicates is critical for any DE analysis.

“While we show that accounting for biological replicates is critical for any DE analysis, this result stresses the paramount importance of addressing this issue in single-cell studies of human tissue.”

Failing to account for biological replicates causes single-cell methods to systematically underestimate the variance of gene expression.

“In contrast, failing to account for biological replicates causes single-cell methods to systematically underestimate the variance of gene expression.”

The inferior performance of single-cell methods can be attributed to their bias towards highly expressed genes.

“Together, these experiments suggest that the inferior performance of single-cell methods can be attributed to their bias towards highly expressed genes.”

The appearance of false discoveries is a universal phenomenon in single-cell differential expression analyses.

“Here, we show that the appearance of false discoveries is a universal phenomenon.”

Accounting for this variability allows pseudobulk methods to correctly identify changes in gene expression caused by a biological perturbation.

“Accounting for this variability allows pseudobulk methods to correctly identify changes in gene expression caused by a biological perturbation.”

Accounting for this variability dictates the biological accuracy of statistical methods.

“Accounting for this variability dictates the biological accuracy of statistical methods.”

The decrease in the variance of gene expression led statistical tests to attribute small changes in gene expression to the effect of the perturbation.

“The decrease in the variance of gene expression led statistical tests to attribute small changes in gene expression to the effect of the perturbation.”

The use of inappropriate statistical methodology can produce false discoveries that compromise the biological interpretation of a single-cell experiment.

“we demonstrate that the use of inappropriate statistical methodology can produce false discoveries that compromise the biological interpretation of a single-cell experiment.”

• AssumptionRests on an assumption: The 18 ground truth datasets are representative of single-cell experiments for assessing DE methods.

Statistical methods for differential expression must account for the intrinsic variability of biological replicates to generate biologically accurate results in single-cell data.

“Statistical methods for differential expression must account for the intrinsic variability of biological replicates to generate biologically accurate results in single-cell data.”

The variability between biological replicates can confound the identification of genes affected by a biological perturbation.

“These experiments demonstrated that the variability between biological replicates can confound the identification of genes affected by a biological perturbation.”

In practice, pseudobulk approaches provide an excellent trade-off between speed and accuracy for single-cell DE analysis.

“These observations suggest that, in practice, pseudobulk approaches provide an excellent trade-off between speed and accuracy for single-cell DE analysis.”

These mechanisms are universal in multifaceted datasets from an additional 46 single-cell RNA-seq studies.

“We demonstrate the universality of these mechanisms in multifaceted datasets from an additional 46 single-cell RNA-seq studies.”

Failing to account for biological replicates induces a bias towards highly expressed genes because these genes are also more variably expressed.

“in partial correlation analyses, only gene expression variance remained correlated with ∆-variance, implying that failing to account for biological replicates induces a bias towards highly expressed genes because these genes are also more variably expressed.”

• AssumptionRests on an assumption: The 46 published scRNA-seq datasets in the compendium are representative of single-cell RNA-seq studies generally.

Failing to account for biological and technical variability makes certain genes disproportionately likely to be spuriously identified as DE.

“we elucidate the progression of mechanisms by which failing to account for biological and technical variability makes certain genes disproportionately likely to be spuriously identified as DE.”

• AssumptionRests on an assumption: The 18 ground truth datasets are representative of single-cell experiments for assessing DE methods.

Single-cell DE methods are poised to produce false discoveries.

“Our results demonstrate that single-cell DE methods are poised to produce false discoveries.”

• AssumptionRests on an assumption: The 18 ground truth datasets are representative of single-cell experiments for assessing DE methods.

Many published findings may be false.

“Our findings suggest that many published findings may be false.”

• AssumptionRests on an assumption: The 18 ground truth datasets are representative of single-cell experiments for assessing DE methods.

The resolution of bulk RNA-seq is insufficient to characterize the multifaceted responses to biological perturbations.

“Consequently, the resolution of bulk RNA-seq is insufficient to characterize the multifaceted responses to biological perturbations.”

False discoveries can be avoided using statistical methodologies that account for between-replicate variation.

“However, we show that false discoveries can be avoided using statistical methodologies that account for between-replicate variation.”

The relative performance of single-cell differential expression methods is contingent on their ability to account for variation between biological replicates.

“Here, we show that the relative performance of these methods is contingent on their ability to account for variation between biological replicates.”

Single-cell methods have a systematic tendency to identify highly expressed genes as DE even when their expression remains unchanged.

“Together, these findings implied a systematic tendency for single-cell methods to identify highly expressed genes as DE, even when their expression remained unchanged.”

The largest decrease in variance for highly expressed genes explains the bias of single-cell methods towards highly expressed genes.

“Moreover, we found that highly expressed genes exhibited the largest decrease in variance in pseudo-replicates, thus explaining the bias of single-cell methods towards highly expressed genes (Supplementary Fig. 4g-k).”

Accounting for variability between biological replicates allows pseudobulk methods to correctly identify changes in gene expression caused by a biological perturbation.

“Accounting for this variability allows pseudobulk methods to correctly identify changes in gene expression caused by a biological perturbation.”

Pseudobulk methods possess a common analytical property that allows them to avoid the bias towards highly expressed genes.

“These findings implied that pseudobulk methods possess a common analytical property that allows them to avoid this bias.”

The same fundamental issues affecting single-cell DE also affect other high-dimensional assays, including spatial transcriptomics.

“Understanding these mechanisms led us to discover that the same fundamental issues affect other high-dimensional assays, including spatial transcriptomics, and are most likely to manifest in studies of human tissue, suggesting that inference at the level of biological replicates is critical to understand the cellular and molecular basis for human disease.”

• AssumptionRests on an assumption: The 46 published scRNA-seq datasets in the compendium are representative of single-cell RNA-seq studies generally.

The best possible approximation to biological ground truth in a scRNA-seq experiment is a matched bulk RNA-seq dataset in the same purified cell type, exposed to the same perturbation under identical experimental conditions, and sequenced in the same laboratory.

“We reasoned that the best possible approximation to the biological ground truth in a scRNA-seq experiment would consist of a matched bulk RNA-seq dataset in the same purified cell type, exposed to the same perturbation under identical experimental conditions, and sequenced in the same laboratory.”

Single-cell DE methods have the potential to produce false discoveries.

“Together, these observations illustrate the potential for single-cell DE methods to produce false discoveries.”

The false-discovery problem in single-cell DE is straightforward to correct with the use of DE methods that account for variability between replicates.

“However, this concerning possibility is straightforward to correct with the use of DE methods that account for variability between replicates.”

• AssumptionRests on an assumption: The 18 ground truth datasets are representative of single-cell experiments for assessing DE methods.

These fundamental issues are most likely to manifest in studies of human tissue.

“Understanding these mechanisms led us to discover that the same fundamental issues affect other high-dimensional assays, including spatial transcriptomics, and are most likely to manifest in studies of human tissue, suggesting that inference at the level of biological replicates is critical to understand the cellular and molecular basis for human disease.”

• AssumptionRests on an assumption: The 46 published scRNA-seq datasets in the compendium are representative of single-cell RNA-seq studies generally.

The appearance of false discoveries in single-cell differential expression is a universal phenomenon.

“Here, we show that the appearance of false discoveries is a universal phenomenon.”

• AssumptionRests on an assumption: The 18 ground truth datasets are representative of single-cell experiments for assessing DE methods.