Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index; 16 of the 16 sources its claims cite were checked for support.

Robust differential expression testing for single-cell CRISPR screens at low multiplicity of infection

Open access

Published version of a preprint

Abstract

Single-cell CRISPR screens (perturb-seq) link genetic perturbations to phenotypic changes in individual cells. The most fundamental task in perturb-seq analysis is to test for association between a perturbation and a count outcome, such as gene expression. We conduct the first-ever comprehensive benchmarking study of association testing methods for low multiplicity-of-infection (MOI) perturb-seq data, finding that existing methods produce excess false positives. We conduct an extensive empirical investigation of the data, identifying three core analysis challenges: sparsity, confounding, and model misspecification. Finally, we develop an association testing method - SCEPTRE low-MOI - that resolves these analysis challenges and demonstrates improved calibration and power.

Overview

CoChat Summary

TL;DR

This paper benchmarks association-testing methods for low-MOI single-cell CRISPR screens and finds that existing methods produce excess false positives due to sparsity, confounding, and model misspecification. It introduces SCEPTRE low-MOI, a negative-binomial score-statistic permutation test that improves calibration and power across datasets.

Key findings

  • In the first comprehensive benchmarking study of six methods on six datasets (five real, one simulated), all existing methods showed miscalibration: the KS test produced over 9,000 false Bonferroni discoveries on the Papalexi data, MAST falsely rejected nearly 2,000 null pairs on the Frangieh IFN- data, and MIMOSCA output p-values strictly less than 0.26 across all pairs.
  • Three core analysis challenges were identified—sparsity, confounding, and model misspecification—and no existing method addressed more than one of them.
  • SCEPTRE low-MOI generally made the fewest false discoveries, averaging 0.7 false discoveries across datasets, roughly a tenfold improvement over the Seurat methods; on the Frangieh IFN- data it made one Bonferroni rejection versus fifteen each for the Seurat methods.
  • SCEPTRE matched or outperformed other methods in power on every dataset, and pairwise QC filtering for effective sample size ≥7 was applied; on standard-sparsity datasets, the authors recommend at least 50–65 cells per perturbation to ensure pairs pass pairwise QC.

Methods

  • Benchmarking: six methods (Seurat-Wilcox, Seurat-NB, t-test, MAST, KS test, MIMOSCA) were applied to six datasets, including five real datasets from Frangieh 2021, Papalexi 2021, and Schraivogel 2020 plus one simulated dataset. Calibration was assessed by constructing null perturbation-gene pairs from non-targeting gRNAs and counting Bonferroni-significant false discoveries; power was assessed on positive control pairs.
  • SCEPTRE low-MOI: for each perturbation-gene pair, gene expression is regressed on perturbation indicators and technical factors via an NB GLM; a GLM score-test z-score is computed, perturbation indicators are permuted B times to generate null z-scores, a skew-normal density is fit to the null z-scores, and a p-value is obtained from the fitted tail. Accelerations include score testing, spectral decomposition for sparse binary treatments, adaptive permutation, skew-normal fitting, and inductive without-replacement sampling.
  • Empirical investigation: exact versus asymptotic Wilcoxon null distributions were compared across effective sample sizes; confounding was examined using biological replicate, NT gRNA presence, and gene expression in the Papalexi data; model misspecification was assessed by applying NB regression to Frangieh IFN- negative control data across effective sample size bins.

Limitations

  • The paper states that SCEPTRE does not currently address off-target effects of some NT gRNAs, ineffective targeting gRNAs, distinguishing direct from indirect perturbation effects, or pooling information across co-expressed gene modules. It also states that evidence for the CAMP robustness property is empirical and a mathematical proof is left to future work.
  • The paper states that GSFA, Normalisr, and scMaGECK were not included in the benchmarking analysis, and that methods with extreme miscalibration on a dataset were excluded from the positive control power analysis.
  • Evident from the text: the calibration check relies on NT gRNA-gene pairs as negative controls, so its validity depends on NT gRNAs being truly null; the paper itself notes that off-targeting NT gRNAs could cause loss of error control. SCEPTRE’s stated validity conditions require either unconfoundedness or correct NB specification up to the size parameter with sufficiently large effective sample size.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13059-024-03254-2

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Comprehensive benchmarking study of single-cell CRISPR screen association testing methods on low-MOI data. a The standard paradigm for association testing on low-MOI single-cell CRISPR screen data. To test for association between a given targeting perturbation and gene, one tests for differential expression of the gene across two groups of cells: those containing the given targeting perturbation and those containing a non-targeting (NT) perturbation. One typically repeats this procedure for a large, preselected set of targeting-perturbation gene pairs, obtaining a discovery set by subjecting the resulting p-values to a multiple comparison correction procedure (e.g., Benjamini-Hochberg). b The calibration check paradigm. One constructs “null” or “negative control” perturbation-gene pairs by coupling each individual NT gRNA to the entire set of genes. One then assesses the calibration of a method by deploying the method to analyze these null pairs. Any p-values that survive the multiple testing correction procedure correspond to false positive discoveries. c, d Results of the calibration check benchmarking analysis on the Papalexi gene expression data. c QQ plot of the null p-values (colored by method) plotted on a negative log transformed scale. Gray region, 95% confidence band. d Number of false discoveries that each method makes on the null pairs after a Bonferroni correction at level 0.1. e, f Similar to panels c–d, but for the Frangieh IFN-γ data
Fig. 1 Comprehensive benchmarking study of single-cell CRISPR screen association testing methods on low-MOI data. a The standard paradigm for association testing on low-MOI single-cell CRISPR screen data. To test for association between a given targeting perturbation and gene, one tests for differential expression of the gene across two groups of cells: those containing the given targeting perturbation and those containing a non-targeting (NT) perturbation. One typically repeats this procedure for a large, preselected set of targeting-perturbation gene pairs, obtaining a discovery set by subjecting the resulting p-values to a multiple comparison correction procedure (e.g., Benjamini-Hochberg). b The calibration check paradigm. One constructs “null” or “negative control” perturbation-gene pairs by coupling each individual NT gRNA to the entire set of genes. One then assesses the calibration of a method by deploying the method to analyze these null pairs. Any p-values that survive the multiple testing correction procedure correspond to false positive discoveries. c, d Results of the calibration check benchmarking analysis on the Papalexi gene expression data. c QQ plot of the null p-values (colored by method) plotted on a negative log transformed scale. Gray region, 95% confidence band. d Number of false discoveries that each method makes on the null pairs after a Bonferroni correction at level 0.1. e, f Similar to panels c–d, but for the Frangieh IFN-γ dataCC BY · PubMed Central
Fig. 2 Sparsity, confounding, and model misspecification are core analysis challenges in single-cell CRISPR screen analysis. a The exact null distribution of the Wilcoxon test statistic (obtained via permutations; gray) on two pairs from the Frangieh IFN-γ data. The Wilcoxon test (and thus Seurat-Wilcox) approximates the exact null distribution using a standard Gaussian density (purple). For pair 1 (left), the Gaussian approximation to the exact null distribution is good (goodness of fit KS statistic = 0.008), while for pair 2 (right) the approximation is inadequate (goodness of fit KS statistic = 0.193). b A plot of pratio (defined as the ratio of the exact Wilcoxon p-value, pexact, to the asymptotic Wilcoxon p-value, pasymptotic) vs. goodness of fit of the Gaussian distribution to the exact null distribution (as quantified by the KS statistic). Each point represents a gene-gRNA pair; pairs 1 and 2 (from panel a) are annotated. As the KS statistic increases (indicating worse fit of the Gaussian distribution to the exact Wilcoxon null distribution), pratio deviates more from one, indicating miscalibration. Points are colored according to the effective sample size of the corresponding pair. c Stratification of the Seurat-Wilcox p-values on the Frangieh IFN-γ negative control data by effective sample size. d An example of confounding on the Papalexi data. Left (resp. right), the fraction of cells that received a given NT gRNA (resp., the relative expression of a given gene) across biological replicates “R1,” “R2,” and “R3.” e Application of NB regression with and without covariates to the Papalexi data. f Stratification of the NB regression p-values on the Papalexi (gene expression) negative control data by effective sample size
Fig. 2 Sparsity, confounding, and model misspecification are core analysis challenges in single-cell CRISPR screen analysis. a The exact null distribution of the Wilcoxon test statistic (obtained via permutations; gray) on two pairs from the Frangieh IFN-γ data. The Wilcoxon test (and thus Seurat-Wilcox) approximates the exact null distribution using a standard Gaussian density (purple). For pair 1 (left), the Gaussian approximation to the exact null distribution is good (goodness of fit KS statistic = 0.008), while for pair 2 (right) the approximation is inadequate (goodness of fit KS statistic = 0.193). b A plot of pratio (defined as the ratio of the exact Wilcoxon p-value, pexact, to the asymptotic Wilcoxon p-value, pasymptotic) vs. goodness of fit of the Gaussian distribution to the exact null distribution (as quantified by the KS statistic). Each point represents a gene-gRNA pair; pairs 1 and 2 (from panel a) are annotated. As the KS statistic increases (indicating worse fit of the Gaussian distribution to the exact Wilcoxon null distribution), pratio deviates more from one, indicating miscalibration. Points are colored according to the effective sample size of the corresponding pair. c Stratification of the Seurat-Wilcox p-values on the Frangieh IFN-γ negative control data by effective sample size. d An example of confounding on the Papalexi data. Left (resp. right), the fraction of cells that received a given NT gRNA (resp., the relative expression of a given gene) across biological replicates “R1,” “R2,” and “R3.” e Application of NB regression with and without covariates to the Papalexi data. f Stratification of the NB regression p-values on the Papalexi (gene expression) negative control data by effective sample sizeCC BY · PubMed Central
Fig. 3 SCEPTRE addresses the core analysis challenges of sparsity, confounding, and model misspecification in theory. a The SCEPTRE algorithm. First, the gene expressions are regressed onto the perturbation indicators and technical factors, and the z-score zobs corresponding to the perturbation indicator is computed. Second, the perturbation indicators are permuted (while the gene expressions and technical factors are held fixed) and the z-score is recomputed, yielding B “null” z-values. Third, a smooth density is fit to the histogram of the null z-values. Fourth, a p-value is computed by evaluating the tail probability of the fitted density at zobs. b A diagram representing the relationship between the variables in the analysis. The technical factors often (but not always) exert a confounding effect on the perturbation indicator and gene expression. c A diagram illustrating the robustness properties of SCEPTRE. The circles represent analysis challenges. A perturbation-gene pair can be affected any subset of the analysis challenges. The color in each region of the diagram indicates whether SCEPTRE is valid on pairs affected by that subset of analysis challenges (blue, yellow, or green = valid; gray = not valid in general). For regions in which SCEPTRE is valid, the color of the region indicates why SCEPTRE is valid (yellow = NB model, blue = permutations). The validity of SCEPTRE is overdetermined on pairs unaffected by any analysis challenge (green region) due to the combination of the NB model and permutations
Fig. 3 SCEPTRE addresses the core analysis challenges of sparsity, confounding, and model misspecification in theory. a The SCEPTRE algorithm. First, the gene expressions are regressed onto the perturbation indicators and technical factors, and the z-score zobs corresponding to the perturbation indicator is computed. Second, the perturbation indicators are permuted (while the gene expressions and technical factors are held fixed) and the z-score is recomputed, yielding B “null” z-values. Third, a smooth density is fit to the histogram of the null z-values. Fourth, a p-value is computed by evaluating the tail probability of the fitted density at zobs. b A diagram representing the relationship between the variables in the analysis. The technical factors often (but not always) exert a confounding effect on the perturbation indicator and gene expression. c A diagram illustrating the robustness properties of SCEPTRE. The circles represent analysis challenges. A perturbation-gene pair can be affected any subset of the analysis challenges. The color in each region of the diagram indicates whether SCEPTRE is valid on pairs affected by that subset of analysis challenges (blue, yellow, or green = valid; gray = not valid in general). For regions in which SCEPTRE is valid, the color of the region indicates why SCEPTRE is valid (yellow = NB model, blue = permutations). The validity of SCEPTRE is overdetermined on pairs unaffected by any analysis challenge (green region) due to the combination of the NB model and permutationsCC BY · PubMed Central
Fig. 4 SCEPTRE demonstrates improved calibration and power relative to existing methods across datasets. a (resp. b) QQ plot of the p-values outputted by Seurat-NB, Seurat-Wilcox, and SCEPTRE on the Frangieh IFN-γ (resp., Papalexi gene expression) negative control data. Gray band, 95% confidence region. c Number of false discoveries (at Bonferroni correction level 0.1) on the negative control data for each method-dataset pair. d Number of true discoveries (significant at level α=10⁻⁵) on the positive control data for each each method-dataset pair
Fig. 4 SCEPTRE demonstrates improved calibration and power relative to existing methods across datasets. a (resp. b) QQ plot of the p-values outputted by Seurat-NB, Seurat-Wilcox, and SCEPTRE on the Frangieh IFN-γ (resp., Papalexi gene expression) negative control data. Gray band, 95% confidence region. c Number of false discoveries (at Bonferroni correction level 0.1) on the negative control data for each method-dataset pair. d Number of true discoveries (significant at level α=10⁻⁵) on the positive control data for each each method-dataset pairCC BY · PubMed Central
Fig. 5 Applying SCEPTRE to make biological discoveries. a The standard workflow involved in applying SCEPTRE to a new dataset, using the Papalexi gene expression data as a running example. First, SCEPTRE is applied to analyze a set of automatically constructed negative control pairs (the “calibration check”). The resulting negative control p-values are plotted on a QQ plot to ensure uniformity (upper left), and the negative control log-fold changes are plotted on a histogram to ensure symmetry about zero (upper right). Second, SCEPTRE is applied to analyze the discovery pairs (the “discovery analysis”). The discovery p-values are superimposed over the negative control p-values to ensure that signal is present in the discovery set (lower left), and a volcano plot is created (lower right). b Computational performance metrics of SCEPTRE on the Frangieh (control) and Papalexi (gene expression) data. A complete trans analysis was conducted on both datasets. Several metrics are reported, including calibration check run time, calibration check peak memory usage, discovery analysis run time, and discovery analysis peak memory usage
Fig. 5 Applying SCEPTRE to make biological discoveries. a The standard workflow involved in applying SCEPTRE to a new dataset, using the Papalexi gene expression data as a running example. First, SCEPTRE is applied to analyze a set of automatically constructed negative control pairs (the “calibration check”). The resulting negative control p-values are plotted on a QQ plot to ensure uniformity (upper left), and the negative control log-fold changes are plotted on a histogram to ensure symmetry about zero (upper right). Second, SCEPTRE is applied to analyze the discovery pairs (the “discovery analysis”). The discovery p-values are superimposed over the negative control p-values to ensure that signal is present in the discovery set (lower left), and a volcano plot is created (lower right). b Computational performance metrics of SCEPTRE on the Frangieh (control) and Papalexi (gene expression) data. A complete trans analysis was conducted on both datasets. Several metrics are reported, including calibration check run time, calibration check peak memory usage, discovery analysis run time, and discovery analysis peak memory usageCC BY · PubMed Central

Verification

Claims the paper makes

10 Valid · 4 Not valid · 0 Not checked · 101 Own results and interpretations

60 more claims · 2 with their cited sources checked

Create a free account to see all 115 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 25
  • It may state two separate claims as one. 32

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 50 references

Research works printed but not linked: 1

  1. Ripley B, Venables B, Bates DM, Hornik K, Gebhardt A, Firth D, et al. Package ‘mass’. Cran R. 2013;538:113–20.

Cited by

See all 28

Metadata

Authors
Barry, T., Mason, K., Roeder, K., & Katsevich, E.
Affiliations
  • Harvard University (US)
  • University of Pennsylvania (US)
  • Carnegie Mellon University (US)
Corresponding author
Katsevich, Eugene
Venue
Genome Biology
Published
2024-05-17
Volume
25
Issue
1
Pages
124-124
ISSN
  • 1465-6906 (ISSN-L)
  • 1474-760X
OpenAlex
W4397012384
Cited by
28
References
50
Citations by year
2026: 13 · 2025: 9 · 2024: 6
FWCI
3.80
Type
journal article
Language
en
Open access
Licence
Locations
  • Genome biology (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Foundation for the National Institutes of Health (R01MH123184)
  • National Science Foundation (DMS-2113072)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › CRISPR and Genetic Engineering
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
Keywords
single-cell CRISPR screens, Perturb-seq, differential expression analysis, low multiplicity of infection, association testing, false positives, sparsity, confounding, model misspecification, scepter, benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Robust differential expression testing for single-cell CRISPR screens at low multiplicity of infection. https://app.cochat.ai/papers/10.1186/s13059-024-03254-2
The paper
Barry, T., Mason, K., Roeder, K., & Katsevich, E. (2024). Robust differential expression testing for single-cell CRISPR screens at low multiplicity of infection. Genome Biology. https://doi.org/10.1186/s13059-024-03254-2

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.