Analytic Pearson residuals for normalization of single-cell RNA-seq UMI data
Open accessPublished version of a preprint
Abstract
Background: Standard preprocessing of single-cell RNA-seq UMI data includes normalization by sequencing depth to remove this technical variability, and nonlinear transformation to stabilize the variance across genes with different expression levels. Instead, two recent papers propose to use statistical count models for these tasks: Hafemeister and Satija (Genome Biol 20:296, 2019) recommend using Pearson residuals from negative binomial regression, while Townes et al. (Genome Biol 20:295, 2019) recommend fitting a generalized PCA model. Here, we investigate the connection between these approaches theoretically and empirically, and compare their effects on downstream processing. Results: We show that the model of Hafemeister and Satija produces noisy parameter estimates because it is overspecified, which is why the original paper employs post hoc smoothing. When specified more parsimoniously, it has a simple analytic solution equivalent to the rank-one Poisson GLM-PCA of Townes et al. Further, our analysis indicates that per-gene overdispersion estimates in Hafemeister and Satija are biased, and that the data are in fact consistent with the overdispersion parameter being independent of gene expression. We then use negative control data without biological variability to estimate the technical overdispersion of UMI counts, and find that across several different experimental protocols, the data are close to Poisson and suggest very moderate overdispersion. Finally, we perform a benchmark to compare the performance of Pearson residuals, variance-stabilizing transformations, and GLM-PCA on scRNA-seq datasets with known ground truth. Conclusions: We demonstrate that analytic Pearson residuals strongly outperform other methods for identifying biologically variable genes, and capture more of the biologically meaningful variation when used for dimensionality reduction.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2
Love, M. I., Huber, W., & Anders, S. (2014) · Genome Biology · doi:10.1186/s13059-014-0550-8 · Cited by 95,106 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Differential expression analysis for sequence count data
Anders, S., & Huber, W. (2010) · Genome Biology · doi:10.1186/gb-2010-11-10-r106 · Cited by 15,581 · see citing works
Comprehensive Integration of Single-Cell Data
Stuart, T., Butler, A., Hoffman, P., Hafemeister, C., Papalexi, E., Mauck, W. M., Hao, Y., Stoeckius, M., Smibert, P., & Satija, R. (2019) · Cell · doi:10.1016/j.cell.2019.05.031 · Cited by 15,453 · see citing works
SCANPY: large-scale single-cell gene expression data analysis
Wolf, F. A., Angerer, P., & Theis, F. J. (2018) · Genome Biology · doi:10.1186/s13059-017-1382-0 · Cited by 8,881 · see citing works
In situ click chemistry generation of cyclooxygenase-2 inhibitors
Bhardwaj, A., Kaur, J., Wuest, M., & Wuest, F. (2017) · Nature Communications · doi:10.1038/s41467-016-0009-6 · Cited by 7,448 · see citing works
Massively parallel digital transcriptional profiling of single cells
Zheng, G. X. Y., Terry, J. M., Belgrader, P., Ryvkin, P., Bent, Z. W., Wilson, R., Ziraldo, S. B., Wheeler, T. D., McDermott, G. P., Zhu, J., et al. (2017) · Nature Communications · doi:10.1038/ncomms14049 · Cited by 7,353 · see citing works
Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets.
Macosko, E. Z., Basu, A., Satija, R., Nemesh, J., Shekhar, K., Goldman, M., Tirosh, I., Bialas, A. R., Kamitaki, N., Martersteck, E. M., et al. (2015) · Cell · doi:10.1016/j.cell.2015.05.002 · Cited by 7,110 · see citing works
Spatial reconstruction of single-cell gene expression data
Satija, R., Farrell, J. A., Gennert, D., Schier, A. F., & Regev, A. (2015) · Nature Biotechnology · doi:10.1038/nbt.3192 · Cited by 7,055 · cited 2× · see citing works
“In brief, these two methods regress out the mean-variance relationship, and return an estimate of the ‘excess’ variance for each gene [34, 35].”
Methodvoom: precision weights unlock linear model analysis tools for RNA-seq read counts
Law, C. W., Chen, Y., Shi, W., & Smyth, G. K. (2014) · Genome Biology · doi:10.1186/gb-2014-15-2-r29 · Cited by 6,155 · see citing works
The single-cell transcriptional landscape of mammalian organogenesis
Cao, J., Spielmann, M., Qiu, X., Huang, X., Ibrahim, D. M., Hill, A. J., Zhang, F., Mundlos, S., Christiansen, L., Steemers, F. J., et al. (2019) · Nature · doi:10.1038/s41586-019-0969-x · Cited by 4,490 · see citing works
Hafemeister, C., & Satija, R. (2019) · Genome Biology · doi:10.1186/s13059-019-1874-1 · Cited by 4,479 · cited 10× · see citing works
“Instead, two recent papers propose to use statistical count models for these tasks: Hafemeister and Satija (2019) recommend using Pearson residuals from negative binomial regression, while Townes et al.”
BackgroundMethodInfluentialDroplet barcoding for single-cell transcriptomics applied to embryonic stem cells
Klein, A. M., Mazutis, L., Akartuna, I., Tallapragada, N., Veres, A., Li, V., Peshkin, L., Weitz, D. A., & Kirschner, M. W. (2015) · Cell · doi:10.1016/j.cell.2015.04.044 · Cited by 3,356 · see citing works
Transformations Related to the Angular and the Square Root
Freeman, M. F., & Tukey, J. W. (1950) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729756 · Cited by 2,634 · see citing works
Deep generative modeling for single-cell transcriptomics
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., & Yosef, N. (2018) · Nature Methods · doi:10.1038/s41592-018-0229-2 · Cited by 2,530 · see citing works
Current best practices in single-cell RNA-seq analysis: a tutorial
Luecken, M. D., & Theis, F. J. (2019) · Molecular Systems Biology · doi:10.15252/msb.20188746 · Cited by 2,231 · see citing works
Correspondence Analysis in Practice
Greenacre, M. J. (2007) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420011234 · Cited by 2,059 · see citing works
Ancient DNA and the rewriting of human history: be sparing with Occam’s razor
Haber, M., Mezzavilla, M., Xue, Y., & Tyler-Smith, C. (2016) · Genome Biology · doi:10.1186/s13059-015-0866-z · Cited by 1,924 · see citing works
Mapping the Mouse Cell Atlas by Microwell-Seq
Han, X., Wang, R., Zhou, Y., Fei, L., Sun, H., Lai, S., Saadatpour, A., Zhou, Z., Chen, H., Ye, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.001 · Cited by 1,597 · see citing works
The transformation of poisson, binomial and negative-binomial data
ANSCOMBE, F. J. (1948) · Biometrika · doi:10.1093/biomet/35.3-4.246 · Cited by 1,300 · see citing works
Comprehensive Classification of Retinal Bipolar Neurons by Single-Cell Transcriptomics
Shekhar, K., Lapan, S. W., Whitney, I. E., Tran, N. M., Macosko, E. Z., Kowalczyk, M., Adiconis, X., Levin, J. Z., Nemesh, J., Goldman, M., et al. (2016) · Cell · doi:10.1016/j.cell.2016.07.054 · Cited by 1,176 · see citing works
The art of using t-SNE for single-cell transcriptomics
Kobak, D., & Berens, P. (2019) · Nature Communications · doi:10.1038/s41467-019-13056-x · Cited by 1,118 · see citing works
Self-citationPooling across cells to normalize single-cell RNA sequencing data with many zero counts
L. Lun, A. T., Bach, K., & Marioni, J. C. (2016) · Genome Biology · doi:10.1186/s13059-016-0947-7 · Cited by 1,108 · see citing works
Orchestrating single-cell analysis with Bioconductor
Amezquita, R. A., Lun, A. T. L., Becht, E., Carey, V. J., Carpp, L. N., Geistlinger, L., Marini, F., Rue-Albrecht, K., Risso, D., Soneson, C., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0654-x · Cited by 945 · see citing works
Correspondence Analysis: A Neglected Multivariate Method
Hill, M. O. (1974) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.2307/2347127 · Cited by 830 · see citing works
Validation of noise models for single-cell transcriptomics
Grün, D., Kester, L., & van Oudenaarden, A. (2014) · Nature Methods · doi:10.1038/nmeth.2930 · Cited by 721 · see citing works
Comparison and evaluation of statistical error models for scRNA-seq
Choudhary, S., & Satija, R. (2022) · Genome Biology · doi:10.1186/s13059-021-02584-9 · Cited by 634 · see citing works
Tran, N. M., Shekhar, K., Whitney, I. E., Jacobi, A., Benhar, I., Hong, G., Yan, W., Adiconis, X., Arnold, M. E., Lee, J. M., et al. (2019) · Neuron · doi:10.1016/j.neuron.2019.11.006 · Cited by 627 · see citing works
Power analysis of single-cell RNA-sequencing experiments
Svensson, V., Natarajan, K. N., Ly, L.-H., Miragaia, R. J., Labalette, C., Macaulay, I. C., Cvejic, A., & Teichmann, S. A. (2017) · Nature Methods · doi:10.1038/nmeth.4220 · Cited by 568 · see citing works
A Connection between Correlation and Contingency
Hirschfeld, H. O. (1935) · Mathematical Proceedings of the Cambridge Philosophical Society · doi:10.1017/s0305004100013517 · Cited by 560 · see citing works
Fast interpolation-based t-SNE for improved visualization of single-cell RNA-seq data
Linderman, G. C., Rachh, M., Hoskins, J. G., Steinerberger, S., & Kluger, Y. (2019) · Nature Methods · doi:10.1038/s41592-018-0308-4 · Cited by 521 · see citing works
Foundations of Linear and Generalized Linear Models
Agresti, A. (2015) · CERN Document Server (European Organization for Nuclear Research) · Cited by 473 · see citing works
Feature selection and dimension reduction for single-cell RNA-Seq based on a multinomial model
Townes, F. W., Hicks, S. C., Aryee, M. J., & Irizarry, R. A. (2019) · Genome Biology · doi:10.1186/s13059-019-1861-6 · Cited by 471 · see citing works
The Geometric Interpretation of Correspondence Analysis
Greenacre, M., & Hastie, T. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478446 · Cited by 455 · see citing works
Yang, L., Li, B., Zheng, X., Li, J., Yang, M., Dong, X., He, G., An, C., & Deng, X. W. (2015) · Nature Communications · doi:10.1038/ncomms9145 · Cited by 426 · see citing works
Lord, D. (2006) · Accident Analysis & Prevention · doi:10.1016/j.aap.2006.02.001 · Cited by 387 · see citing works
Choudhuri, D., Srinivasan, S. G., Gibson, M. A., Zheng, Y., Jaeger, D. L., Fraser, H. L., & Banerjee, R. (2017) · Nature Communications · doi:10.1038/s41467-017-02112-z · Cited by 384 · see citing works
Droplet scRNA-seq is not zero-inflated
Svensson, V. (2019) · doi:10.1101/582064 · Cited by 356 · see citing works
glmGamPoi: fitting Gamma-Poisson generalized linear models on single cell count data
Ahlmann-Eltze, C., & Huber, W. K. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa1009 · Cited by 330 · see citing works
The Transformation of Poisson, Binomial and Negative-Binomial Data
Anscombe, F. J. (1948) · Biometrika · doi:10.2307/2332343 · Cited by 304 · see citing works
A systematic performance evaluation of clustering methods for single-cell RNA-seq data
Duò, A., Robinson, M. D., & Soneson, C. (2018) · F1000Research · doi:10.12688/f1000research.15666.2 · Cited by 296 · see citing works
Kim, J. K., Kolodziejczyk, A. A., Ilicic, T., Teichmann, S. A., & Marioni, J. C. (2015) · Nature Communications · doi:10.1038/ncomms9687 · Cited by 251 · see citing works
Lord, D., & Miranda-Moreno, L. (2007) · Safety Science · doi:10.1016/j.ssci.2007.03.005 · Cited by 242 · see citing works
Sarkar, A., & Stephens, M. (2021) · Nature Genetics · doi:10.1038/s41588-021-00873-4 · Cited by 224 · see citing works
A systematic performance evaluation of clustering methods for single-cell RNA-seq data
Duò, A., Robinson, M. D., & Soneson, C. (2020) · F1000Research · doi:10.12688/f1000research.15666.3 · Cited by 200 · see citing works
Interpretable factor models of single-cell RNA-seq via variational autoencoders
Svensson, V., Gayoso, A., Yosef, N., & Pachter, L. S. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa169 · Cited by 192 · see citing works
Imaging and writing magnetic domains in the non-collinear antiferromagnet Mn3Sn
Reichlová, H., Janda, T., Godinho, J., Μάρκου, Α., Kriegner, D., Schlitz, R., Železný, J., Šobáň, Z., Bejarano, M., Schultheiß, H., et al. (2019) · Nature Communications · doi:10.1038/s41467-019-13391-z · Cited by 187 · see citing works
Efficient Algorithms for t-distributed Stochastic Neighborhood EmbeddingPreprint
Linderman, G. C., Rachh, M., Hoskins, J. G., Steinerberger, S., & Kluger, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.09005 · Cited by 172 · see citing works
Comparison of Transformations for Single-Cell RNA-Seq Data
Ahlmann-Eltze, C., & Huber, W. (2021) · doi:10.1101/2021.06.24.449781 · Cited by 148 · see citing works
Gene expression distribution deconvolution in single-cell RNA sequencing
Wang, J., Huang, M. C., Torre, E. A., Dueck, H., Shaffer, S. M., Murray, J. I., Raj, A., Li, M., & Zhang, N. R. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1721085115 · Cited by 121 · see citing works
Estimation of the Negative Binomial Parameter κ by Maximum Quasi -Likelihood
Clark, S. J., & PERRY, J. N. (1989) · Biometrics · doi:10.2307/2532055 · Cited by 110 · see citing works
Eling, N., Richard, A. C., Richardson, S., Marioni, J. C., & Vallejos, C. A. (2018) · Cell Systems · doi:10.1016/j.cels.2018.06.011 · Cited by 99 · see citing works
Bayesian inference of gene expression states from single-cell RNA-seq data
Breda, J., Zavolan, M., & van Nimwegen, E. (2021) · Nature Biotechnology · doi:10.1038/s41587-021-00875-x · Cited by 87 · see citing works
Why You Cannot Transform Your Way out of Trouble for Small Counts
Warton, D. I. (2017) · Biometrics · doi:10.1111/biom.12728 · Cited by 57 · see citing works
Lun, A. T. L. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/404962 · Cited by 53 · see citing works
Multivariate data analysis: The French way
Holmes, S. (2008) · Institute of Mathematical Statistics eBooks · doi:10.1214/193940307000000455 · Cited by 47 · see citing works
Attraction-Repulsion Spectrum in Neighbor EmbeddingsPreprint
Böhm, J. N., Berens, P., & Kobak, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.08902 · Cited by 31 · see citing works
Self-citationAnalytic Pearson residuals for normalization of single-cell RNA-seq UMI dataPreprint
Lause, J., Berens, P., & Kobak, D. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.12.01.405886 · Cited by 28 · see citing works
Self-citationBar‐Lev, S. K., & Enis, P. (1988) · Biometrika · doi:10.1093/biomet/75.4.803 · Cited by 24 · see citing works
A Unifying Perspective on Neighbor Embeddings along the Attraction-Repulsion SpectrumPreprint
Böhm, J. N., Berens, P., & Kobak, D. (2021) · arXiv (Cornell University) · Cited by 24 · see citing works
Self-citationWillson, L. J., Folks, J. L., & Young, J. H. (1986) · Metrika · doi:10.1007/bf01894768 · Cited by 23 · see citing works
baredSC: Bayesian approach to retrieve expression distribution of single-cell data
Lopez‐Delisle, L., & Delisle, J.-B. (2022) · BMC Bioinformatics · doi:10.1186/s12859-021-04507-8 · Cited by 10 · see citing works
Straightforward clustering of single-cell RNA-Seq data with t-SNE and DBSCANPreprint
Wagner, F. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/770388 · Cited by 9 · see citing works
Comparison and evaluation of statistical error models for scRNA-seqPreprint
Choudhary, S., & Satija, R. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.07.07.451498 · Cited by 8 · see citing works
Wagner, F. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.06.08.140673 · Cited by 5 · see citing works
baredSC: Bayesian Approach to Retrieve Expression Distribution of Single-CellPreprint
Lopez‐Delisle, L., & Delisle, J.-B. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.05.26.445740 · Cited by 3 · see citing works
berenslab/umi-normalization: Submission v3
Lause, J. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5150534 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Best practices for single-cell analysis across modalities
Heumos, L., Schaar, A. C., Lance, C., Litinetskaya, A., Drost, F., Zappia, L., Lücken, M. D., Strobl, D. C., Henao, J., Curion, F., et al. (2023) · Nature Reviews Genetics · doi:10.1038/s41576-023-00586-w · Cited by 1,051 · see citing works
“…was shown to work better for stabilizing variance for subsequent dimensionality reduction 33 , Scran performs well for batch correction tasks 34,35 , and analytical Pearson residuals are better suited for selection of biologically variable genes and identification of rare cell identities 31 .”
MethodComparison and evaluation of statistical error models for scRNA-seq
Choudhary, S., & Satija, R. (2022) · Genome Biology · doi:10.1186/s13059-021-02584-9 · Cited by 634 · see citing works
Pathogenic variants damage cell composition and single cell transcription in cardiomyopathies
Reichart, D., Lindberg, E. L., Maatz, H., Miranda, A. M. A., Viveiros, A., Shvetsov, N., Gärtner, A., Nadelmann, E. R., Lee, M., Kanemaru, K., et al. (2022) · Science · doi:10.1126/science.abo1984 · Cited by 240 · see citing works
Normalizing and denoising protein expression data from droplet-based single cell profiling
Mulè, M. P., Martins, A. J., & Tsang, J. S. (2022) · Nature Communications · doi:10.1038/s41467-022-29356-8 · Cited by 197 · see citing works
Comparison of Transformations for Single-Cell RNA-Seq Data
Ahlmann-Eltze, C., & Huber, W. (2021) · doi:10.1101/2021.06.24.449781 · Cited by 148 · see citing works
Normalization for sampled count data
Booeshaghi, A. S., Hallgrímsdóttir, I. B., Gálvez-Merchán, Á., & Pachter, L. (2022) · doi:10.1101/2022.05.06.490859 · Cited by 39 · see citing works
Barry, T., Mason, K., Roeder, K., & Katsevich, E. (2024) · Genome Biology · doi:10.1186/s13059-024-03254-2 · Cited by 28 · see citing works
Gamache, J., Gingerich, D., Shwab, E. K., Barrera, J., Garrett, M. E., Hume, C., Crawford, G. E., Ashley-Koch, A. E., & Chiba-Falek, O. (2023) · Cell & Bioscience · doi:10.1186/s13578-023-01120-5 · Cited by 20 · see citing works
- Authors
- Lause, J., Berens, P., & Kobak, D.
- Affiliations
- University of Tübingen (DE)
- Bernstein Center for Computational Neuroscience Tübingen (DE)
- Venue
- Genome Biology
- Published
- 2021-09-06
- Volume
- 22
- Issue
- 1
- Pages
- 258-258
- ISSN
- 1465-6906 (ISSN-L)
- 1474-760X
- PMC
- PMC8419999
- OpenAlex
- W3196992636
- Cited by
- 195
- References
- 67
- Citations by year
- 2026: 42 · 2025: 41 · 2024: 63 · 2023: 27 · 2022: 18 · 2021: 4
- FWCI
- 9.18
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Genome biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Deutsche Forschungsgemeinschaft (BE5601/4-1, BE5601/6-1, EXC 2064 ML 390727645)
- Bundesministerium f?r Bildung und Forschung (01GQ1601, 01IS18039A)
- National Institute of Mental Health (1U19MH114830-01)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- single-cell RNA sequencing, UMI counts, data normalization, Pearson residuals, negative binomial regression, overdispersion, variance stabilizing transformation, sequencing depth, technical variability, dimensionality reduction, count data models
- MeSH terms
- RNA-Seq, Algorithms, Regression Analysis, Retina, Sequence Analysis, RNA, Principal Component Analysis, Databases, Genetic, Organogenesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
