CoChat

Accuracy, robustness and scalability of dimensionality reduction methods for single-cell RNA-seq analysis

Open access
Sun, Shiquan, Zhu, Jiaqiang, Ma, Ying, Zhou, Xiang · 2019 · Genome biology

Published version of a preprint

The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index.

Abstract

BACKGROUND: Dimensionality reduction is an indispensable analytic component for many areas of single-cell RNA sequencing (scRNA-seq) data analysis. Proper dimensionality reduction can allow for effective noise removal and facilitate many downstream analyses that include cell clustering and lineage reconstruction. Unfortunately, despite the critical importance of dimensionality reduction in scRNA-seq analysis and the vast number of dimensionality reduction methods developed for scRNA-seq studies, few comprehensive comparison studies have been performed to evaluate the effectiveness of different dimensionality reduction methods in scRNA-seq. RESULTS: We aim to fill this critical knowledge gap by providing a comparative evaluation of a variety of commonly used dimensionality reduction methods for scRNA-seq studies. Specifically, we compare 18 different dimensionality reduction methods on 30 publicly available scRNA-seq datasets that cover a range of sequencing techniques and sample sizes. We evaluate the performance of different dimensionality reduction methods for neighborhood preserving in terms of their ability to recover features of the original expression matrix, and for cell clustering and lineage reconstruction in terms of their accuracy and robustness. We also evaluate the computational scalability of different dimensionality reduction methods by recording their computational cost. CONCLUSIONS: Based on the comprehensive evaluation results, we provide important guidelines for choosing dimensionality reduction methods for scRNA-seq data analysis. We also provide all analysis scripts used in the present study at www.xzlab.org/reproduce.html.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13059-019-1898-6

Cites

91 of 91 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Overview of the evaluation workflow for dimensionality reduction methods. We obtained a total of 30 publicly available scRNA-seq data from GEO and 10X Genomics website. We also simulated two addition simulation data sets. For each of the 32 data sets in turn, we applied 18 dimensionality reduction methods to extract the low-dimensional components. Afterwards, we evaluated the performance of dimensionality reduction methods by evaluating how effective the low-dimensional components extracted from dimensionality reduction methods are for downstream analysis. We did so by evaluating the two commonly applied downstream analysis: clustering analysis and lineage reconstruction analysis. In the analysis, we varied the number of low-dimensional components extracted from these dimensionality reduction methods. The performance of each dimensionality reduction method is qualified by Jaccard index for neighborhood preserving, normalized mutual information (NMI) and adjusted rand index (ARI) for cell clustering analysis, and Kendall correlation coefficient for trajectory inference. We also recorded the stability of each dimensionality reduction method across data splits and recorded the computation time for each dimensionality reduction method. Through the comprehensive evaluation, we eventually provide practical guidelines for practitioners to choose dimensionality reduction methods for scRNA-seq data analysis
Fig. 1 Overview of the evaluation workflow for dimensionality reduction methods. We obtained a total of 30 publicly available scRNA-seq data from GEO and 10X Genomics website. We also simulated two addition simulation data sets. For each of the 32 data sets in turn, we applied 18 dimensionality reduction methods to extract the low-dimensional components. Afterwards, we evaluated the performance of dimensionality reduction methods by evaluating how effective the low-dimensional components extracted from dimensionality reduction methods are for downstream analysis. We did so by evaluating the two commonly applied downstream analysis: clustering analysis and lineage reconstruction analysis. In the analysis, we varied the number of low-dimensional components extracted from these dimensionality reduction methods. The performance of each dimensionality reduction method is qualified by Jaccard index for neighborhood preserving, normalized mutual information (NMI) and adjusted rand index (ARI) for cell clustering analysis, and Kendall correlation coefficient for trajectory inference. We also recorded the stability of each dimensionality reduction method across data splits and recorded the computation time for each dimensionality reduction method. Through the comprehensive evaluation, we eventually provide practical guidelines for practitioners to choose dimensionality reduction methods for scRNA-seq data analysisCC BY · PubMed Central
Fig. 2 Dimensionality reduction method performance evaluated by k-means clustering based on NMI in downstream cell clustering analysis. We compared 18 dimensionality reduction methods (columns), including factor analysis (FA), principal component analysis (PCA), independent component analysis (ICA), Diffusion Map, nonnegative matrix factorization (NMF), Poisson NMF, zero-inflated factor analysis (ZIFA), zero-inflated negative binomial based wanted variation extraction (ZINB-WaVE), probabilistic count matrix factorization (pCMF), deep count autoencoder network (DCA), scScope, generalized linear model principal component analysis (GLMPCA), multidimensional scaling (MDS), locally linear embedding (LLE), local tangent space alignment (LTSA), Isomap, uniform manifold approximation and projection (UMAP), and t-distributed stochastic neighbor embedding (tSNE). We evaluated their performance on 14 real scRNA-seq data sets (UMI-based data are labeled as purple; non-UMI-based data are labeled as blue) and 2 simulated data sets (rows). The simulated data based on Kumar data is labeled with #. The performance of each dimensionality reduction method is measured by normalized mutual information (NMI). For each data set, we compared the four different numbers of low-dimensional components. The four numbers equal to 0.5%, 1%, 2%, and 3% of the total number of cells in big data and equal to 2, 6, 14, and 20 in small data (which are labeled with*). For convenience, we only listed 0.5%, 1%, 2%, and 3% on x-axis. No results for ICA are shown in the table (gray fills) because ICA cannot handle the large number of features in that data. No results for LTSA are shown (gray fills) because error occurred when we applied the clustering method on LTSA extracted low-dimensional components there. Note that, for tSNE, we only extracted two low-dimensional components due to the limitation of the tSNE software
Fig. 2 Dimensionality reduction method performance evaluated by k-means clustering based on NMI in downstream cell clustering analysis. We compared 18 dimensionality reduction methods (columns), including factor analysis (FA), principal component analysis (PCA), independent component analysis (ICA), Diffusion Map, nonnegative matrix factorization (NMF), Poisson NMF, zero-inflated factor analysis (ZIFA), zero-inflated negative binomial based wanted variation extraction (ZINB-WaVE), probabilistic count matrix factorization (pCMF), deep count autoencoder network (DCA), scScope, generalized linear model principal component analysis (GLMPCA), multidimensional scaling (MDS), locally linear embedding (LLE), local tangent space alignment (LTSA), Isomap, uniform manifold approximation and projection (UMAP), and t-distributed stochastic neighbor embedding (tSNE). We evaluated their performance on 14 real scRNA-seq data sets (UMI-based data are labeled as purple; non-UMI-based data are labeled as blue) and 2 simulated data sets (rows). The simulated data based on Kumar data is labeled with #. The performance of each dimensionality reduction method is measured by normalized mutual information (NMI). For each data set, we compared the four different numbers of low-dimensional components. The four numbers equal to 0.5%, 1%, 2%, and 3% of the total number of cells in big data and equal to 2, 6, 14, and 20 in small data (which are labeled with*). For convenience, we only listed 0.5%, 1%, 2%, and 3% on x-axis. No results for ICA are shown in the table (gray fills) because ICA cannot handle the large number of features in that data. No results for LTSA are shown (gray fills) because error occurred when we applied the clustering method on LTSA extracted low-dimensional components there. Note that, for tSNE, we only extracted two low-dimensional components due to the limitation of the tSNE softwareCC BY · PubMed Central
Fig. 3 Dimensionality reduction method performance evaluated by Kendall correlation in the downstream trajectory inference analysis. We compared 17 dimensionality reduction methods (columns), including factor analysis (FA), principal component analysis (PCA), independent component analysis (ICA), Diffusion Map, nonnegative matrix factorization (NMF), Poisson NMF, zero-inflated factor analysis (ZIFA), zero-inflated negative binomial-based wanted variation extraction (ZINB-WaVE), probabilistic count matrix factorization (pCMF), deep count autoencoder network (DCA), generalized linear model principal component analysis (GLMPCA), multidimensional scaling (MDS), locally linear embedding (LLE), local tangent space alignment (LTSA), Isomap, uniform manifold approximation and projection (UMAP), and t-distributed stochastic neighbor embedding (tSNE). We evaluated their performance on 14 real scRNA-seq data sets (rows) in terms of lineage inference accuracy. We used Slingshot with k-means as the initial step for lineage inference. The performance of each dimensionality reduction method is measured by Kendall correlation. For each data set, we compared four different numbers of low-dimensional components (2, 6, 14, and 20; four sub-columns under each column). Gray fills in the table represents missing results where Slingshot gave out errors when we supplied the extracted low-dimensional components from the corresponding dimensionality reduction method. Note that, for tSNE, we only extracted two low-dimensional components due to the limitation of the tSNE software
Fig. 3 Dimensionality reduction method performance evaluated by Kendall correlation in the downstream trajectory inference analysis. We compared 17 dimensionality reduction methods (columns), including factor analysis (FA), principal component analysis (PCA), independent component analysis (ICA), Diffusion Map, nonnegative matrix factorization (NMF), Poisson NMF, zero-inflated factor analysis (ZIFA), zero-inflated negative binomial-based wanted variation extraction (ZINB-WaVE), probabilistic count matrix factorization (pCMF), deep count autoencoder network (DCA), generalized linear model principal component analysis (GLMPCA), multidimensional scaling (MDS), locally linear embedding (LLE), local tangent space alignment (LTSA), Isomap, uniform manifold approximation and projection (UMAP), and t-distributed stochastic neighbor embedding (tSNE). We evaluated their performance on 14 real scRNA-seq data sets (rows) in terms of lineage inference accuracy. We used Slingshot with k-means as the initial step for lineage inference. The performance of each dimensionality reduction method is measured by Kendall correlation. For each data set, we compared four different numbers of low-dimensional components (2, 6, 14, and 20; four sub-columns under each column). Gray fills in the table represents missing results where Slingshot gave out errors when we supplied the extracted low-dimensional components from the corresponding dimensionality reduction method. Note that, for tSNE, we only extracted two low-dimensional components due to the limitation of the tSNE softwareCC BY · PubMed Central
Fig. 4 The computation time (in hours) for different dimensionality reduction methods. We recorded computing time for 18 dimensionality reduction methods on simulated data sets with a varying number of low-dimensional components and a varying number of sample sizes. Compared dimensionality reduction methods include factor analysis (FA; light green), principal component analysis (PCA; light blue), independent component analysis (ICA; blue), Diffusion Map (pink), nonnegative matrix factorization (NMF; green), Poisson NMF(light orange), zero-inflated factor analysis (ZIFA; light pink), zero-inflated negative binomial based wanted variation extraction (ZINB-WaVE; orange), probabilistic count matrix factorization (pCMF; light purple), deep count autoencoder network (DCA; yellow), scScope (purple), generalized linear model principal component analysis (GLMPCA; red), multidimensional scaling (MDS; cyan), locally linear embedding (LLE; blue green), local tangent space alignment (LTSA; teal blue), Isomap (gray), uniform manifold approximation and projection (UMAP; brown), and t-distributed stochastic neighbor embedding (tSNE; dark red). a Computation time for different dimensionality reduction methods (y-axis) changes with respect to an increasing number of low-dimensional components (x-axis). The number of cells is fixed to be 500 and the number of genes is fixed to be 10,000 in this set of simulations. Three methods (ZINB-WaVE, pCMF, and ZIFA) become noticeably computationally more expensive than the remaining methods with increasing number of low-dimensional components. b Computation time for different dimensionality reduction methods (y-axis) changes with respect to an increasing sample size (i.e., the number of cells) in the data. Computing time is recorded on a single thread of an Intel Xeon E5-2683 2.00-GHz processor. The number of low-dimensional components is fixed to be 22 in this set of simulations for most methods, except for tSNE which used two low-dimensional components due to the limitation of the tSNE software. Note that some methods are implemented with parallelization capability (e.g., ZINB-WaVE and pCMF) though we tested them on a single thread for fair comparison across methods. Note that PCA is similar to ICA in a and scScope is similar to several other efficient methods in b; thus, their lines may appear to be missing. Overall, three methods (ZIFA, pCMF, and ZINB-WaVE) become noticeably computationally more expensive than the remaining methods with increasing number of cells in the data
Fig. 4 The computation time (in hours) for different dimensionality reduction methods. We recorded computing time for 18 dimensionality reduction methods on simulated data sets with a varying number of low-dimensional components and a varying number of sample sizes. Compared dimensionality reduction methods include factor analysis (FA; light green), principal component analysis (PCA; light blue), independent component analysis (ICA; blue), Diffusion Map (pink), nonnegative matrix factorization (NMF; green), Poisson NMF(light orange), zero-inflated factor analysis (ZIFA; light pink), zero-inflated negative binomial based wanted variation extraction (ZINB-WaVE; orange), probabilistic count matrix factorization (pCMF; light purple), deep count autoencoder network (DCA; yellow), scScope (purple), generalized linear model principal component analysis (GLMPCA; red), multidimensional scaling (MDS; cyan), locally linear embedding (LLE; blue green), local tangent space alignment (LTSA; teal blue), Isomap (gray), uniform manifold approximation and projection (UMAP; brown), and t-distributed stochastic neighbor embedding (tSNE; dark red). a Computation time for different dimensionality reduction methods (y-axis) changes with respect to an increasing number of low-dimensional components (x-axis). The number of cells is fixed to be 500 and the number of genes is fixed to be 10,000 in this set of simulations. Three methods (ZINB-WaVE, pCMF, and ZIFA) become noticeably computationally more expensive than the remaining methods with increasing number of low-dimensional components. b Computation time for different dimensionality reduction methods (y-axis) changes with respect to an increasing sample size (i.e., the number of cells) in the data. Computing time is recorded on a single thread of an Intel Xeon E5-2683 2.00-GHz processor. The number of low-dimensional components is fixed to be 22 in this set of simulations for most methods, except for tSNE which used two low-dimensional components due to the limitation of the tSNE software. Note that some methods are implemented with parallelization capability (e.g., ZINB-WaVE and pCMF) though we tested them on a single thread for fair comparison across methods. Note that PCA is similar to ICA in a and scScope is similar to several other efficient methods in b; thus, their lines may appear to be missing. Overall, three methods (ZIFA, pCMF, and ZINB-WaVE) become noticeably computationally more expensive than the remaining methods with increasing number of cells in the dataCC BY · PubMed Central
Fig. 5 Practical guideline for choosing dimensionality reduction methods in scRNA-seq analysis. Compared dimensionality reduction methods include factor analysis (FA), principal component analysis (PCA), independent component analysis (ICA), Diffusion Map, nonnegative matrix factorization (NMF), Poisson NMF, zero-inflated factor analysis (ZIFA), zero-inflated negative binomial-based wanted variation extraction (ZINB-WaVE), probabilistic count matrix factorization (pCMF), deep count autoencoder network (DCA), scScope, generalized linear model principal component analysis (GLMPCA), multidimensional scaling (MDS), locally linear embedding (LLE), local tangent space alignment (LTSA), Isomap, uniform manifold approximation and projection (UMAP), and t-distributed stochastic neighbor embedding (tSNE). The count-based methods are colored in purple while non-count-based methods are colored in blue. Methods are ranked by their average performance across the criteria from left to right. The performance is colored and numerically coded: good performance = 2 (sky blue), intermediate performance = 1 (orange), and poor performance = 0 (gray)
Fig. 5 Practical guideline for choosing dimensionality reduction methods in scRNA-seq analysis. Compared dimensionality reduction methods include factor analysis (FA), principal component analysis (PCA), independent component analysis (ICA), Diffusion Map, nonnegative matrix factorization (NMF), Poisson NMF, zero-inflated factor analysis (ZIFA), zero-inflated negative binomial-based wanted variation extraction (ZINB-WaVE), probabilistic count matrix factorization (pCMF), deep count autoencoder network (DCA), scScope, generalized linear model principal component analysis (GLMPCA), multidimensional scaling (MDS), locally linear embedding (LLE), local tangent space alignment (LTSA), Isomap, uniform manifold approximation and projection (UMAP), and t-distributed stochastic neighbor embedding (tSNE). The count-based methods are colored in purple while non-count-based methods are colored in blue. Methods are ranked by their average performance across the criteria from left to right. The performance is colored and numerically coded: good performance = 2 (sky blue), intermediate performance = 1 (orange), and poor performance = 0 (gray)CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 91 references

Research works printed but not linked: 1

  1. Sun S, Zhu J, Ma Y, Zhou X: Accuracy, robustness and scalability of dimensionality reduction methods for single-cell RNA-seq analysis. Anal Code Github Repository 2019. https://github.com/xzhoulab/DRComparison. Accessed 13 Oct 2019. PMC6902413 31823809

Cited by

See all 228

Metadata

Authors
Sun, S., Zhu, J., Ma, Y., & Zhou, X.
Affiliations
  • Northwestern Polytechnical University (CN)
  • University of Michigan (US)
Corresponding author
Zhou, Xiang
Venue
Genome biology
Published
2019-12
Volume
20
Issue
1
Pages
269-269
ISSN
  • 1465-6906 (ISSN-L)
  • 1474-760X
OpenAlex
W2996274328
Cited by
225
References
90
Citations by year
2026: 20 · 2025: 27 · 2024: 37 · 2023: 34 · 2022: 36 · 2021: 51 · 2020: 18 · 2019: 2
FWCI
8.85
Type
journal article
Language
en
Open access
Licence
Locations
  • Genome biology (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Deep Blue (University of Michigan) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • the National Institutes of Health (R01HG009124, R01GM126553, R01HD088558, U01HL137182)
  • the National Science Foundation (DMS1712933)
  • the Chan Zuckerberg Initiative DAF (2018-181314)
  • the National Natural Science Foundation of China (61902319)
  • the Natural Science Foundation of Shaanxi Province (2019JQ127)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
Keywords
dimensionality reduction, single-cell RNA sequencing, method comparison, neighborhood preservation, cell clustering, lineage reconstruction, computational scalability, robustness, noise reduction, reproducibility
MeSH terms
Cluster Analysis, Sequence Analysis, RNA, Single-Cell Analysis, Dimensional Measurement Accuracy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Accuracy, robustness and scalability of dimensionality reduction methods for single-cell RNA-seq analysis. https://app.cochat.ai/papers/10.1186/s13059-019-1898-6
The paper
Sun, S., Zhu, J., Ma, Y., & Zhou, X. (2019). Accuracy, robustness and scalability of dimensionality reduction methods for single-cell RNA-seq analysis. Genome biology. https://doi.org/10.1186/s13059-019-1898-6

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.