CoChat

Normalization and variance stabilization of single-cell RNA-seq data using regularized negative binomial regression

Open access
Hafemeister, Christoph, Satija, Rahul · 2019 · Genome Biology

Published version of a preprint

The DOI matches its Crossref record; 41 of the 42 references it cites were found in a registry or an index; 23 of the 27 sources its claims cite were checked for support.

Abstract

Single-cell RNA-seq (scRNA-seq) data exhibits significant cell-to-cell variation due to technical factors, including the number of molecules detected in each cell, which can confound biological heterogeneity with technical effects. To address this, we present a modeling framework for the normalization and variance stabilization of molecular count data from scRNA-seq experiments. We propose that the Pearson residuals from “regularized negative binomial regression,” where cellular sequencing depth is utilized as a covariate in a generalized linear model, successfully remove the influence of technical characteristics from downstream analyses while preserving biological heterogeneity. Importantly, we show that an unconstrained negative binomial model may overfit scRNA-seq data, and overcome this by pooling information across genes with similar abundances to obtain stable parameter estimates. Our procedure omits the need for heuristic steps including pseudocount addition or log-transformation and improves common downstream analytical tasks such as variable gene selection, dimensional reduction, and differential expression. Our approach can be applied to any UMI-based scRNA-seq dataset and is freely available as part of the R package , with a direct interface to our single-cell toolkit .

Overview

CoChat Summary

TL;DR

The paper introduces regularized negative binomial regression for normalizing UMI-based single-cell RNA-seq data, using Pearson residuals as variance-stabilized expression values. It argues that this removes sequencing-depth effects while preserving biological heterogeneity better than log-normalization or unconstrained negative binomial models.

Key findings

  • A single scaling factor does not normalize all genes equally: in a 33,148-cell PBMC dataset with median 1891 UMI/cell and 16,809 genes detected in ≥5 cells, log-normalization effectively normalized only low/medium-abundance genes, while high-abundance genes remained depth-confounded and low-depth cells showed disproportionately high variance for high-abundance genes.
  • Unconstrained negative binomial regression overfit scRNA-seq data, especially for low/medium-abundance genes, producing irreproducible parameter estimates across bootstraps; kernel-regularized parameters were consistent across subsamples.
  • Pearson residuals from regularized NB regression had minimal correlation with sequencing depth, residual mean centered at 0, residual variance peaked at 1, and high-variance outlier genes were known immune cell markers such as PPBP, GNLY, and IGJ.
  • In differential expression tests, splitting CD14+ monocytes and downsampling one group to 50% depth yielded >2000 DE genes with log-normalized data at FDR 0.01 but only 11 genes with Pearson residuals; downsampling CD16+ monocytes to 20% depth gave highly concordant Pearson-residual DE results, unlike log-normalized data.

Methods

  • For each gene, UMI counts were modeled with a generalized linear model using total cell UMI count as a covariate, a negative binomial error distribution, and a log link, estimating intercept, slope, and dispersion θ.
  • The workflow has three steps: independent per-gene NB regression; kernel regression (ksmooth, normal kernel, bandwidth from bw.SJ multiplied by a default bandwidth adjustment factor of 3) to regularize parameters across genes with similar mean expression; then a constrained NB regression/affine transformation to Pearson residuals, clipped to a maximum of √N.
  • Analyses used five UMI datasets from plate- and droplet-based protocols, focusing on a 33,148-cell 10x PBMC dataset, plus a Chromium control dataset, a 26,439 murine bipolar cell batch-extension example, simulation, PCA/UMAP, and t-test differential expression with Benjamini-Hochberg FDR.

Limitations

  • The paper states that the method assumes most genes do not exhibit significant biological variation, which may be overly simplistic for highly heterogeneous samples; it suggests a possible pre-clustering extension to reduce this concern.
  • The batch-covariate extension requires all cell types to share a similar batch effect; the paper notes that cell-type-specific batch effects or shifts in cell-type proportions require non-linear batch correction strategies.
  • The paper states that kernel bandwidth selection matters, with extreme values causing over- or under-smoothing; it also reports 11 false-positive DE genes attributed to highly expressed genes that are difficult to fit because few genes share similar mean values. Evident from the text, the method is presented for UMI-based scRNA-seq and is not validated here for non-UMI or other single-cell modalities.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13059-019-1874-1

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 33,148 PBMC dataset from 10X Genomics. a Distribution of total UMI counts / cell (“sequencing depth”). b We placed genes into six groups, based on their average expression in the dataset. c For each gene group, we examined the average relationship between observed counts and cell sequencing depth. We fit a smooth line for each gene individually and combined results based on the groupings in b. Black line shows mean, colored region indicates interquartile range. d Same as in c, but showing scaled log-normalized values instead of UMI counts. Values were scaled (z-scored) so that a single Y-axis range could be used. e Relationship between gene variance and cell sequencing depth; cells were placed into five equal-sized groups based on total UMI counts (group 1 has the greatest depth), and we calculated the total variance of each gene group within each bin. For effectively normalized data, each cell bin should contribute 20% to the variance of each gene group
Fig. 1 33,148 PBMC dataset from 10X Genomics. a Distribution of total UMI counts / cell (“sequencing depth”). b We placed genes into six groups, based on their average expression in the dataset. c For each gene group, we examined the average relationship between observed counts and cell sequencing depth. We fit a smooth line for each gene individually and combined results based on the groupings in b. Black line shows mean, colored region indicates interquartile range. d Same as in c, but showing scaled log-normalized values instead of UMI counts. Values were scaled (z-scored) so that a single Y-axis range could be used. e Relationship between gene variance and cell sequencing depth; cells were placed into five equal-sized groups based on total UMI counts (group 1 has the greatest depth), and we calculated the total variance of each gene group within each bin. For effectively normalized data, each cell bin should contribute 20% to the variance of each gene groupCC BY · PubMed Central
Fig. 2 We fit NB regression models for each gene individually and bootstrapped the process to measure uncertainty in the resulting parameter estimates. a Model parameters for 16,809 genes for the NB regression model, plotted as a function of average gene abundance across the 33,148 cells. The color of each point indicates a parameter uncertainty score as determined by bootstrapping (“Methods” section). Pink line shows the regularized parameters obtained via kernel regression. b Standard deviation (σ) of NB regression model parameters across multiple bootstraps. Red points: σ for unconstrained NB model. Blue points: σ for regularized NB model, which is substantially reduced in comparison. Black trendline shows an increase in σ for low-abundance genes, highlighting the potential for overfitting in the absence of regularization
Fig. 2 We fit NB regression models for each gene individually and bootstrapped the process to measure uncertainty in the resulting parameter estimates. a Model parameters for 16,809 genes for the NB regression model, plotted as a function of average gene abundance across the 33,148 cells. The color of each point indicates a parameter uncertainty score as determined by bootstrapping (“Methods” section). Pink line shows the regularized parameters obtained via kernel regression. b Standard deviation (σ) of NB regression model parameters across multiple bootstraps. Red points: σ for unconstrained NB model. Blue points: σ for regularized NB model, which is substantially reduced in comparison. Black trendline shows an increase in σ for low-abundance genes, highlighting the potential for overfitting in the absence of regularizationCC BY · PubMed Central
Fig. 3 Pearson residuals from regularized NB regression represent effectively normalized scRNA-seq data. Panels a and b are analogous to Fig. 1 d and e, but calculated using Pearson residuals. c Boxplot of Pearson correlations between Pearson residuals and total cell UMI counts for each of the six gene bins. All three panels demonstrate that in contrast to log-normalized data, the level and variance of Pearson residuals is independent of sequencing depth
Fig. 3 Pearson residuals from regularized NB regression represent effectively normalized scRNA-seq data. Panels a and b are analogous to Fig. 1 d and e, but calculated using Pearson residuals. c Boxplot of Pearson correlations between Pearson residuals and total cell UMI counts for each of the six gene bins. All three panels demonstrate that in contrast to log-normalized data, the level and variance of Pearson residuals is independent of sequencing depthCC BY · PubMed Central
Fig. 4 Regularized NB regression removes variation due to sequencing depth, but retains biological heterogeneity. a Distribution of residual mean, across all genes, is centered at 0. b Density of residual gene variance peaks at 1, as would be expected when the majority of genes do not vary across cell types. c Variance of Pearson residuals is independent of gene abundance, demonstrating that the GLM has successfully captured the mean-variance relationship inherent in the data. Genes with high residual variance are exclusively cell-type markers. d In contrast to a regularized NB, a Poisson error model does not fully capture the variance in highly expressed genes. An unconstrained (non-regularized) NB model overfits scRNA-seq data, attributing almost all variation to technical effects. As a result, even cell-type markers exhibit low residual variance. Mean-variance trendline shown in blue for each panel
Fig. 4 Regularized NB regression removes variation due to sequencing depth, but retains biological heterogeneity. a Distribution of residual mean, across all genes, is centered at 0. b Density of residual gene variance peaks at 1, as would be expected when the majority of genes do not vary across cell types. c Variance of Pearson residuals is independent of gene abundance, demonstrating that the GLM has successfully captured the mean-variance relationship inherent in the data. Genes with high residual variance are exclusively cell-type markers. d In contrast to a regularized NB, a Poisson error model does not fully capture the variance in highly expressed genes. An unconstrained (non-regularized) NB model overfits scRNA-seq data, attributing almost all variation to technical effects. As a result, even cell-type markers exhibit low residual variance. Mean-variance trendline shown in blue for each panelCC BY · PubMed Central
Fig. 5 The regularized NB model is an attractive middle ground between two extremes. a For four genes, we show the relationship between cell sequencing depth and molecular counts. White points show the observed data. Background color represents the Pearson residual magnitude under three error models. For MALAT1 (does not vary across cell types), the Poisson error model does not account for overdispersion and incorrectly infers significant residual variation (biological heterogeneity). For S100A9 (a CD14+ monocyte marker) and CD74 (expressed in antigen-presenting cells), the non-regularized NB model overfits the data and collapses biological heterogeneity. For PPBP (a Megakaryocyte marker), both non-regularized models wrongly fit a negative slope. b Boxplot of Pearson residuals for models shown in a. X-axis range shown is limited to [ − 8, 25] for visual clarity
Fig. 5 The regularized NB model is an attractive middle ground between two extremes. a For four genes, we show the relationship between cell sequencing depth and molecular counts. White points show the observed data. Background color represents the Pearson residual magnitude under three error models. For MALAT1 (does not vary across cell types), the Poisson error model does not account for overdispersion and incorrectly infers significant residual variation (biological heterogeneity). For S100A9 (a CD14+ monocyte marker) and CD74 (expressed in antigen-presenting cells), the non-regularized NB model overfits the data and collapses biological heterogeneity. For PPBP (a Megakaryocyte marker), both non-regularized models wrongly fit a negative slope. b Boxplot of Pearson residuals for models shown in a. X-axis range shown is limited to [ − 8, 25] for visual clarityCC BY · PubMed Central
Fig. 6 Downstream analyses of Pearson residuals are unaffected by differences in sequencing depth. a UMAP embedding of the 33,148 cell PBMC dataset using either log-normalization or Pearson residuals. Both normalization schemes lead to similar results with respect to the major and minor cell populations in the dataset. However, in analyses of log-normalized data, cells within a cluster are ordered along a gradient that is correlated with sequencing depth. b Within the four major cell types, the percent of variance explained by sequencing depth under both normalization schemes. c UMAP embedding of two groups of biologically identical CD14+ monocytes, where one group was randomly downsampled to 50% depth. d Results of differential expression (DE) test between the two groups shown in c. Gray areas indicate expected group mean difference by chance and a false discovery rate cutoff of 1%. e Results of DE test between CD14+ and CD16+ monocytes, before and after randomly downsampling the CD16+ cells to 20% depth
Fig. 6 Downstream analyses of Pearson residuals are unaffected by differences in sequencing depth. a UMAP embedding of the 33,148 cell PBMC dataset using either log-normalization or Pearson residuals. Both normalization schemes lead to similar results with respect to the major and minor cell populations in the dataset. However, in analyses of log-normalized data, cells within a cluster are ordered along a gradient that is correlated with sequencing depth. b Within the four major cell types, the percent of variance explained by sequencing depth under both normalization schemes. c UMAP embedding of two groups of biologically identical CD14+ monocytes, where one group was randomly downsampled to 50% depth. d Results of differential expression (DE) test between the two groups shown in c. Gray areas indicate expected group mean difference by chance and a false discovery rate cutoff of 1%. e Results of DE test between CD14+ and CD16+ monocytes, before and after randomly downsampling the CD16+ cells to 20% depthCC BY · PubMed Central

Verification

Claims the paper makes

16 Valid · 1 Not valid · 2 Not checked · 80 Own results and interpretations

47 more claims · 1 with their cited sources checked

Create a free account to see all 99 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 39
  • Probably not a substantial claim. 15

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 41 references

Research works printed but not linked: 1

  1. McInnes L, Healy J, Saul N, Grossberger L. UMAP: uniform manifold approximation and projection. J Open Source Softw. 2018; 3(29):861. · DOI · Paper page

Cited by

See all 4,576

Metadata

Authors
Hafemeister, C., & Satija, R.
Affiliations
New York Genome Center (US)
Venue
Genome Biology
Published
2019-12
Volume
20
Issue
1
Pages
296-296
ISSN
  • 1465-6906 (ISSN-L)
  • 1474-760X
OpenAlex
W2997280954
Cited by
4,479
References
42
Citations by year
2026: 622 · 2025: 795 · 2024: 897 · 2023: 773 · 2022: 675 · 2021: 460 · 2020: 231 · 2019: 24
FWCI
116.50
Type
journal article
Language
en
Open access
Licence
Locations
  • Genome biology (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • New York State Department of Health
  • California Institute of Technology
  • New York Genome Center
  • Deutsche Forschungsgemeinschaft (328558384)
  • National Institutes of Health (1dp2hg009623-01, OT2OD026673)
  • National Institute of Mental Health (R01MH071679)
  • National Human Genome Research Institute (DP2-HG-009623)
  • National Institute of Neurological Disorders and Stroke (R35NS097404)
  • Eunice Kennedy Shriver National Institute of Child Health and Human Development (R01HD096770)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
Keywords
single-cell RNA sequencing, data normalization, variance stabilization, Pearson residuals, sequencing depth, technical variability, overfitting, differential gene expression, Seurat
MeSH terms
Humans, Regression Analysis, Software, Sequence Analysis, RNA, Single-Cell Analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Normalization and variance stabilization of single-cell RNA-seq data using regularized negative binomial regression. https://app.cochat.ai/papers/10.1186/s13059-019-1874-1
The paper
Hafemeister, C., & Satija, R. (2019). Normalization and variance stabilization of single-cell RNA-seq data using regularized negative binomial regression. Genome Biology. https://doi.org/10.1186/s13059-019-1874-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.