A systematic performance evaluation of clustering methods for single-cell RNA-seq data
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Subpopulation identification, usually via some form of unsupervised clustering, is a fundamental step in the analysis of many single-cell RNA-seq data sets. This has motivated the development and application of a broad range of clustering methods, based on various underlying algorithms. Here, we provide a systematic and extensible performance evaluation of 12 clustering algorithms, including both methods developed explicitly for scRNA-seq data and more general-purpose methods. The methods were evaluated using 9 publicly available scRNA-seq data sets as well as three simulations with varying degree of cluster separability. The same feature selection approaches were used for all methods, allowing us to focus on the investigation of the performance of the clustering algorithms themselves. We evaluated the ability of recovering known subpopulations, the stability and the run time of the methods. Additionally, we investigated whether the performance could be improved by generating consensus partitions from multiple individual clustering methods. We found substantial differences in the performance, run time and stability between the methods, with SC3 and Seurat showing the most favorable results. Additionally, we found that consensus clustering typically did not improve the performance compared to the best of the combined methods, but that several of the top-performing methods already perform some type of consensus clustering. The R scripts providing an extensible framework for the evaluation of new methods and data sets are available on GitHub ( https://github.com/markrobinsonuzh/scRNAseq_clustering_comparison ).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523
Use of Ranks in One-Criterion Variance Analysis
Kruskal, W. H., & Wallis, W. A. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483441 · Cited by 13,912 · cited 1×
“We also investigated whether the number of detected features per cell differed between the clusters, using a Kruskal-Wallis test 49 .”
MethodNear-optimal probabilistic RNA-seq quantification
Bray, N. L., Pimentel, H., Melsted, P., & Pachter, L. (2016) · Nature Biotechnology · doi:10.1038/nbt.3519 · Cited by 9,723
Hubert, L., & Arabie, P. (1985) · Journal of Classification · doi:10.1007/bf01908075 · Cited by 8,219 · cited 1×
“In order to evaluate how well the inferred clusters recovered the true subpopulations, we used the Hubert-Arabie Adjusted Rand Index (ARI) for comparing two partitions 46 .”
MethodMassively parallel digital transcriptional profiling of single cells
Zheng, G. X. Y., Terry, J. M., Belgrader, P., Ryvkin, P., Bent, Z. W., Wilson, R., Ziraldo, S. B., Wheeler, T. D., McDermott, G. P., Zhu, J., et al. (2017) · Nature Communications · doi:10.1038/ncomms14049 · Cited by 7,353
Trapnell, C., Cacchiarelli, D., Grimsby, J., Pokharel, P., Li, S., Morse, M., Lennon, N. J., Livak, K. J., Mikkelsen, T. S., & Rinn, J. L. (2014) · Nature Biotechnology · doi:10.1038/nbt.2859 · Cited by 7,314
Spatial reconstruction of single-cell gene expression data
Satija, R., Farrell, J. A., Gennert, D., Schier, A. F., & Regev, A. (2015) · Nature Biotechnology · doi:10.1038/nbt.3192 · Cited by 7,055
mRNA-Seq whole-transcriptome analysis of a single cell
Tang, F., Barbacioru, C., Wang, Y., Nordman, E., Lee, C., Xu, N., Wang, X., Bodeau, J., Tuch, B. B., Siddiqui, A., et al. (2009) · Nature Methods · doi:10.1038/nmeth.1315 · Cited by 3,969
Droplet barcoding for single-cell transcriptomics applied to embryonic stem cells
Klein, A. M., Mazutis, L., Akartuna, I., Tallapragada, N., Veres, A., Li, V., Peshkin, L., Weitz, D. A., & Kirschner, M. W. (2015) · Cell · doi:10.1016/j.cell.2015.04.044 · Cited by 3,356
Research works printed but not linked: 30
- E Macosko, 2015, Highly Parallel Genome-wide Expression Profiling of Individual Cells Using Nanoliter Droplets., Cell., 161, 1202-1214, 10.1016/j.cell.2015.05.002 · DOI · Paper page
- V Svensson, 2018, Exponential scaling of single-cell RNA-seq in the past decade., Nat Protoc., 13, 599-604, 10.1038/nprot.2017.149 · DOI · Paper page
- C Ziegenhain, 2018, Quantitative single-cell transcriptomics., Brief Funct Genomics., ely009, 10.1093/bfgp/ely009 · DOI
- R Bacher, 2016, Design and computational analysis of single-cell RNA-sequencing experiments., Genome Biol., 17, 63, 10.1186/s13059-016-0927-y · DOI · Paper page
- N Aghaeepour, 2013, Critical assessment of automated flow cytometry data analysis techniques., Nat Methods., 10, 228-238, 10.1038/nmeth.2365 · DOI · Paper page
- L Weber, 2016, Comparison of clustering methods for high-dimensional single-cell flow and mass cytometry data., Cytometry A., 89, 1084-1096, 10.1002/cyto.a.23030 · DOI · Paper page
- V Menon, 2017, Clustering single cells: a review of approaches on high-and low-depth single-cell RNA-seq data., Brief Funct Genomics., elx044, 10.1093/bfgp/elx044 · DOI
- P Langfelder, 2008, WGCNA: an R package for weighted correlation network analysis., BMC Bioinformatics., 9, 559, 10.1186/1471-2105-9-559 · DOI · Paper page
- S Freytag, 2017, Cluster headache: Comparing clustering tools for 10X single cell sequencing data., bioRxiv., 10.1101/203752 · DOI
- T Andrews, 2018, Identifying cell populations with scRNASeq., Mol Aspects Med., 59, 114-122, 10.1016/j.mam.2017.07.002 · DOI · Paper page
- P Koh, 2016, An atlas of transcriptional, chromatin accessibility, and surface marker changes in human mesoderm development., Sci Data., 3, 10.1038/sdata.2016.109 · DOI
- M Ramos, 2017, Software for the integration of Multi-Omics experiments in Bioconductor., bioRxiv., 10.1101/144774 · DOI
- V Ntranos, 2016, Fast and accurate single-cell RNA-Seq analysis by clustering of transcript-compatibility counts., Genome Biol., 17, 112, 10.1186/s13059-016-0970-8 · DOI
- C Soneson, 2018, Towards unified quality verification of synthetic count data with countsimQC., Bioinformatics., 34, 691-692, 10.1093/bioinformatics/btx631 · DOI · Paper page
- K Pearson, 1901, On lines and planes of closest fit to systems of points in space., Philos Mag., 2, 559-572, 10.1080/14786440109462720 · DOI · Paper page
- L van der Maaten, 2008, Visualizing data using t-SNE., J Mach Learn Res., 9, 2579-2605
- T Andrews, 2018, Dropout-based feature selection for scRNASeq., bioRxiv., 10.1101/065094 · DOI
- A Senabouth, 2017, ascend: R package for analysis of single cell RNA-seq data., bioRxiv., 10.1101/207704 · DOI
- P Lin, 2017, CIDR: Ultrafast and accurate clustering through imputation for single-cell RNA-seq data., Genome Biol., 18, 59, 10.1186/s13059-017-1188-0 · DOI · Paper page
- S Van Gassen, 2015, Flowsom: Using self-organizing maps for visualization and interpretation of cytometry data., Cytometry A., 87, 636-645, 10.1002/cyto.a.22625 · DOI · Paper page
- J Ward, 1963, Hierarchical grouping to optimize an objective function., J Am Stat Assoc., 58, 236-244, 10.1080/01621459.1963.10500845 · DOI · Paper page
- J Žurauskienė, 2016, pcaReduce: hierarchical clustering of single cell transcriptional profiles., BMC Bioinformatics., 17, 140, 10.1186/s12859-016-0984-y · DOI · Paper page
- L Van Der Maaten, 2014, Accelerating t-SNE using tree-based algorithms., J Mach Learn Res., 15, 1-21
- Y Yang, 2017, SAFE-clustering: Single-cell aggregated (from ensemble) clustering for single-cell RNA-seq data., bioRxiv., 10.1101/215723 · DOI
- Z Ji, 2016, TSCAN: Pseudo-time reconstruction and evaluation in single-cell RNA-seq analysis., Nucleic Acids Res., 44, e117, 10.1093/nar/gkw430 · DOI · Paper page
- C Shannon, 1948, A mathematical theory of communication., Bell Syst Tech J., 27, 379-423, 10.1002/j.1538-7305.1948.tb01338.x · DOI · Paper page
- K Hornik, 2005, A CLUE for CLUster Ensembles., J Stat Softw., 14, 1-25, 10.18637/jss.v014.i12 · DOI · Paper page
- U Von Luxburg, 2010, Clustering stability: an overview., Foundations and Trends in Machine Learning., 2, 235-274, 10.1561/2200000008 · DOI · Paper page
- D Severson, 2018, BEARscc determines robustness of single-cell clusters using simulated technical replicates., Nat Commun., 9, 10.1038/s41467-018-03608-y · DOI
- A Duò, 2018, markrobinsonuzh/scRNAseq_clustering_comparison: F1000 v1 (Version 0.9)., Zenodo.
Cited by
Current best practices in single-cell RNA-seq analysis: a tutorial
Luecken, M. D., & Theis, F. J. (2019) · Molecular Systems Biology · doi:10.15252/msb.20188746 · Cited by 2,231
Single‐cell RNA sequencing technologies and applications: A brief overview
Jovic, D., Liang, X., Zeng, H., Lin, L., Xu, F., & Luo, Y. (2022) · Clinical and Translational Medicine · doi:10.1002/ctm2.694 · Cited by 1,271
Spatially informed cell-type deconvolution for spatial transcriptomics
Ma, Y., & Zhou, X. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01273-7 · Cited by 516
Feature selection and dimension reduction for single-cell RNA-Seq based on a multinomial model
Townes, F. W., Hicks, S. C., Aryee, M. J., & Irizarry, R. A. (2019) · Genome Biology · doi:10.1186/s13059-019-1861-6 · Cited by 471
Embracing the dropouts in single-cell RNA-seq analysis
Qiu, P. (2020) · Nature Communications · doi:10.1038/s41467-020-14976-9 · Cited by 391
Metadata
- Authors
- Duò, A., Robinson, M. D., & Soneson, C.
- Venue
- F1000Research
- Publisher
- F1000 Research Ltd
- Published
- 2018-07-26
- Volume
- 7
- Pages
- 1141
- ISSN
- 2046-1402 (ISSN-L)
- OpenAlex
- W4238363316
- Cited by
- 93
- References
- 23
- FWCI
- 4.71
- Type
- journal article
- Language
- en
- Update notices
- Open access
- gold
- Funders
- Chan Zuckerberg Initiative (182828)
- Swiss National Science Foundation (310030_175841)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A systematic performance evaluation of clustering methods for single-cell RNA-seq data. https://app.cochat.ai/papers/10.12688/f1000research.15666.1
- The paper
- Duò, A., Robinson, M. D., & Soneson, C. (2018). A systematic performance evaluation of clustering methods for single-cell RNA-seq data. F1000Research. https://doi.org/10.12688/f1000research.15666.1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.