A systematic performance evaluation of clustering methods for single-cell RNA-seq data
Open accessAbstract
Subpopulation identification, usually via some form of unsupervised clustering, is a fundamental step in the analysis of many single-cell RNA-seq data sets. This has motivated the development and application of a broad range of clustering methods, based on various underlying algorithms. Here, we provide a systematic and extensible performance evaluation of 14 clustering algorithms implemented in R, including both methods developed explicitly for scRNA-seq data and more general-purpose methods. The methods were evaluated using nine publicly available scRNA-seq data sets as well as three simulations with varying degree of cluster separability. The same feature selection approaches were used for all methods, allowing us to focus on the investigation of the performance of the clustering algorithms themselves. We evaluated the ability of recovering known subpopulations, the stability and the run time and scalability of the methods. Additionally, we investigated whether the performance could be improved by generating consensus partitions from multiple individual clustering methods. We found substantial differences in the performance, run time and stability between the methods, with SC3 and Seurat showing the most favorable results. Additionally, we found that consensus clustering typically did not improve the performance compared to the best of the combined methods, but that several of the top-performing methods already perform some type of consensus clustering. All the code used for the evaluation is available on GitHub ( https://github.com/markrobinsonuzh/scRNAseq_clustering_comparison ). In addition, an R package providing access to data and clustering results, thereby facilitating inclusion of new methods and data sets, is available from Bioconductor ( https://bioconductor.org/packages/DuoClustering2018 ).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Community detection based on graph modularity optimization via the Louvain algorithm works best for cluster identification.
“Independent benchmarks5,50,51 showed that community detection based on graph modularity optimization via the Louvain algorithm works best for cluster identification.”
From Best practices for single-cell analysis across modalities (2023) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486 · see citing works
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421 · see citing works
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
WGCNA: an R package for weighted correlation network analysis
Langfelder, P., & Horvath, S. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-559 · Cited by 27,252 · see citing works
Hierarchical Grouping to Optimize an Objective Function
Ward, J. H. (1963) · Journal of the American Statistical Association · doi:10.1080/01621459.1963.10500845 · Cited by 21,341 · see citing works
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523 · see citing works
Use of Ranks in One-Criterion Variance Analysis
Kruskal, W. H., & Wallis, W. A. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483441 · Cited by 13,912 · see citing works
LIII. On lines and planes of closest fit to systems of points in space
Karl Pearson, F. R. S. (1901) · The London Edinburgh and Dublin Philosophical Magazine and Journal of Science · doi:10.1080/14786440109462720 · Cited by 13,658 · see citing works
Near-optimal probabilistic RNA-seq quantification
Bray, N. L., Pimentel, H., Melsted, P., & Pachter, L. (2016) · Nature Biotechnology · doi:10.1038/nbt.3519 · Cited by 9,723 · see citing works
Hubert, L., & Arabie, P. (1985) · Journal of Classification · doi:10.1007/bf01908075 · Cited by 8,219 · see citing works
Massively parallel digital transcriptional profiling of single cells
Zheng, G. X. Y., Terry, J. M., Belgrader, P., Ryvkin, P., Bent, Z. W., Wilson, R., Ziraldo, S. B., Wheeler, T. D., McDermott, G. P., Zhu, J., et al. (2017) · Nature Communications · doi:10.1038/ncomms14049 · Cited by 7,353 · see citing works
Trapnell, C., Cacchiarelli, D., Grimsby, J., Pokharel, P., Li, S., Morse, M., Lennon, N. J., Livak, K. J., Mikkelsen, T. S., & Rinn, J. L. (2014) · Nature Biotechnology · doi:10.1038/nbt.2859 · Cited by 7,314 · see citing works
Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets.
Macosko, E. Z., Basu, A., Satija, R., Nemesh, J., Shekhar, K., Goldman, M., Tirosh, I., Bialas, A. R., Kamitaki, N., Martersteck, E. M., et al. (2015) · Cell · doi:10.1016/j.cell.2015.05.002 · Cited by 7,110 · see citing works
Spatial reconstruction of single-cell gene expression data
Satija, R., Farrell, J. A., Gennert, D., Schier, A. F., & Regev, A. (2015) · Nature Biotechnology · doi:10.1038/nbt.3192 · Cited by 7,055 · see citing works
Reversed graph embedding resolves complex single-cell trajectories
Qiu, X., Mao, Q., Tang, Y., Wang, L., Chawla, R., Pliner, H. A., & Trapnell, C. (2017) · Nature Methods · doi:10.1038/nmeth.4402 · Cited by 5,070 · see citing works
mRNA-Seq whole-transcriptome analysis of a single cell
Tang, F., Barbacioru, C., Wang, Y., Nordman, E., Lee, C., Xu, N., Wang, X., Bodeau, J., Tuch, B. B., Siddiqui, A., et al. (2009) · Nature Methods · doi:10.1038/nmeth.1315 · Cited by 3,969 · see citing works
Droplet barcoding for single-cell transcriptomics applied to embryonic stem cells
Klein, A. M., Mazutis, L., Akartuna, I., Tallapragada, N., Veres, A., Li, V., Peshkin, L., Weitz, D. A., & Kirschner, M. W. (2015) · Cell · doi:10.1016/j.cell.2015.04.044 · Cited by 3,356 · see citing works
Cell types in the mouse cortex and hippocampus revealed by single-cell RNA-seq
Zeisel, A., Muñoz-Manchado, A. B., Codeluppi, S., Lönnerberg, P., La Manno, G., Juréus, A., Marques, S., Munguba, H., He, L., Betsholtz, C., et al. (2015) · Science · doi:10.1126/science.aaa1934 · Cited by 3,124 · see citing works
Use of Ranks in One-Criterion Variance Analysis
Kruskal, W., & Wallis, W. A. (1952) · Journal of the American Statistical Association · doi:10.2307/2280779 · Cited by 2,455 · see citing works
Smart-seq2 for sensitive full-length transcriptome profiling in single cells
Picelli, S., Björklund, Å. K., Faridani, O. R., Sagasser, S., Winberg, G., & Sandberg, R. (2013) · Nature Methods · doi:10.1038/nmeth.2639 · Cited by 2,358 · see citing works
Accelerating t-SNE using tree-based algorithms
van der Maaten, L. (2014) · Journal of machine learning research · doi:10.5555/2627435.2697068 · Cited by 2,207 · see citing works
FlowSOM: Using self‐organizing maps for visualization and interpretation of cytometry data
Gassen, S. V., Callebaut, B., van Helden, M. J. G., Lambrecht, B. N., Demeester, P. M. A., Dhaene, T., & Saeys, Y. (2015) · Cytometry Part A · doi:10.1002/cyto.a.22625 · Cited by 1,930 · see citing works
McCarthy, D. J., Campbell, K. R., Lun, A. T. L., & Wills, Q. F. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btw777 · Cited by 1,852 · see citing works
SC3: consensus clustering of single-cell RNA-seq data
Kiselev, V. Y., Kirschner, K., Schaub, M. T., Andrews, T., Yiu, A., Chandra, T., Natarajan, K. N., Reik, W., Barahona, M., Green, A. R., et al. (2017) · Nature Methods · doi:10.1038/nmeth.4236 · Cited by 1,500 · see citing works
Pooling across cells to normalize single-cell RNA sequencing data with many zero counts
L. Lun, A. T., Bach, K., & Marioni, J. C. (2016) · Genome Biology · doi:10.1186/s13059-016-0947-7 · Cited by 1,108 · see citing works
Exponential scaling of single-cell RNA-seq in the past decade
Svensson, V., Vento‐Tormo, R., & Teichmann, S. A. (2018) · Nature Protocols · doi:10.1038/nprot.2017.149 · Cited by 936 · see citing works
Splatter: simulation of single-cell RNA sequencing data
Zappia, L., Phipson, B., & Oshlack, A. (2017) · Genome Biology · doi:10.1186/s13059-017-1305-0 · Cited by 905 · see citing works
Bias, robustness and scalability in single-cell differential expression analysis
Soneson, C., & Robinson, M. D. (2018) · Nature Methods · doi:10.1038/nmeth.4612 · Cited by 770 · see citing works
Self-citationValidation of noise models for single-cell transcriptomics
Grün, D., Kester, L., & van Oudenaarden, A. (2014) · Nature Methods · doi:10.1038/nmeth.2930 · Cited by 721 · see citing works
Missing data and technical variability in single-cell RNA-sequencing experiments
Hicks, S. C., Townes, F. W., Teng, M., & Irizarry, R. A. (2018) · Biostatistics · doi:10.1093/biostatistics/kxx053 · Cited by 635 · see citing works
TSCAN: Pseudo-time reconstruction and evaluation in single-cell RNA-seq analysis
Ji, Z., & Ji, H. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw430 · Cited by 624 · see citing works
Critical assessment of automated flow cytometry data analysis techniques
Aghaeepour, N., Finak, G., Hoos, H. H., Mosmann, T. R., Brinkman, R. R., Gottardo, R., & Scheuermann, R. H. (2013) · Nature Methods · doi:10.1038/nmeth.2365 · Cited by 589 · see citing works
De Novo Prediction of Stem Cell Identity using Single-Cell Transcriptome Data
Grün, D., Muraro, M. J., Boisset, J.-C., Wiebrands, K., Lyubimova, A., Dharmadhikari, G., van den Born, M., van Es, J., Jansen, E., Clevers, H., et al. (2016) · Cell Stem Cell · doi:10.1016/j.stem.2016.05.010 · Cited by 586 · see citing works
Power analysis of single-cell RNA-sequencing experiments
Svensson, V., Natarajan, K. N., Ly, L.-H., Miragaia, R. J., Labalette, C., Macaulay, I. C., Cvejic, A., & Teichmann, S. A. (2017) · Nature Methods · doi:10.1038/nmeth.4220 · Cited by 568 · see citing works
CIDR: Ultrafast and accurate clustering through imputation for single-cell RNA-seq data
Lin, P., Troup, M., & Ho, J. W. K. (2017) · Genome biology · doi:10.1186/s13059-017-1188-0 · Cited by 529 · see citing works
Design and computational analysis of single-cell RNA-sequencing experiments
Bacher, R. L., & Kendziorski, C. M. (2016) · Genome biology · doi:10.1186/s13059-016-0927-y · Cited by 437 · see citing works
Comparison of clustering methods for high‐dimensional single‐cell flow and mass cytometry data
Weber, L. M., & Robinson, M. D. (2016) · Cytometry Part A · doi:10.1002/cyto.a.23030 · Cited by 381 · see citing works
Self-citationDeconstructing transcriptional heterogeneity in pluripotent stem cells
Kumar, R. M., Cahan, P., Shalek, A. K., Satija, R., Jay DaleyKeyser, A., Li, H., Zhang, J., Pardee, K., Gennert, D., Trombetta, J. J., et al. (2014) · Nature · doi:10.1038/nature13920 · Cited by 356 · see citing works
Batch effects and the effective design of single-cell gene expression studies
Tung, P.-Y., Blischak, J. D., Hsiao, C. J., Knowles, D. A., Burnett, J. E., Pritchard, J. K., & Gilad, Y. (2017) · Scientific Reports · doi:10.1038/srep39921 · Cited by 356 · see citing works
Exploring the single-cell RNA-seq analysis landscape with the scRNA-tools database
Zappia, L., Phipson, B., & Oshlack, A. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006245 · Cited by 304 · see citing works
Clustering Stability: An Overview
von Luxburg, U. (2010) · Foundations and Trends® in Machine Learning · doi:10.48550/arxiv.1007.1075 · Cited by 302 · see citing works
pcaReduce: hierarchical clustering of single cell transcriptional profiles
Žurauskienė, J., & Yau, C. (2016) · BMC Bioinformatics · doi:10.1186/s12859-016-0984-y · Cited by 298 · see citing works
M3Drop: dropout-based feature selection for scRNASeq
Andrews, T. S., & Hemberg, M. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty1044 · Cited by 283 · see citing works
Hornik, K. (2005) · Journal of Statistical Software · doi:10.18637/jss.v014.i12 · Cited by 268 · see citing works
Identifying cell populations with scRNASeq
Andrews, T. S., & Hemberg, M. (2017) · Molecular Aspects of Medicine · doi:10.1016/j.mam.2017.07.002 · Cited by 240 · see citing works
Comparison of clustering tools in R for medium-sized 10x Genomics single-cell RNA-sequencing data
Freytag, S., Tian, L., Lönnstedt, I., Ng, M., & Bahlo, M. (2018) · F1000Research · doi:10.12688/f1000research.15809.1 · Cited by 220 · see citing works
Exponential scaling of single-cell RNA-seq in the last decadePreprint
Svensson, V., Tormo, R. V., & Teichmann, S. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.01379 · Cited by 166 · see citing works
Comparison of clustering tools in R for medium-sized 10x Genomics single-cell RNA-sequencing data
Freytag, S., Tian, L., Lönnstedt, I. M., et al. (2018) · F1000Research · doi:10.12688/f1000research.15809.2 · Cited by 159 · see citing works
Lember, J., & Koloydenko, A. A. (2014) · Journal of Machine Learning Research · Cited by 130 · see citing works
SAFE-clustering: Single-cell Aggregated (from Ensemble) clustering for single-cell RNA-seq data
Yang, Y., Huh, R., Culpepper, H., Lin, Y., Love, M. I., & Li, Y. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty793 · Cited by 125 · see citing works
Fast and accurate single-cell RNA-seq analysis by clustering of transcript-compatibility counts
Ntranos, V., Kamath, G. M., Zhang, J. M., Pachter, L. S., & Tse, D. N. C. (2016) · Genome biology · doi:10.1186/s13059-016-0970-8 · Cited by 107 · see citing works
Software for the Integration of Multiomics Experiments in Bioconductor
Ramos, M., Schiffer, L., Re, A., Azhar, R., Basunia, A., Cabrera, C. R., Chan, T., Chapman, P. J., Davis, S. R., Gómez-Cabrero, D., et al. (2017) · Cancer Research · doi:10.1158/0008-5472.can-17-0344 · Cited by 107 · see citing works
Clustering single cells: a review of approaches on high-and low-depth single-cell RNA-seq data
Menon, V. (2017) · Briefings in Functional Genomics · doi:10.1093/bfgp/elx044 · Cited by 73 · see citing works
Koh, P. W., Sinha, R., Barkal, A. A., Morganti, R. M., Chen, A. C., Weissman, I. L., Ang, L. T., Kundaje, A. B., & Loh, K. M. (2016) · Scientific Data · doi:10.1038/sdata.2016.109 · Cited by 72 · see citing works
Towards unified quality verification of synthetic count data with countsimQC
Soneson, C., & Robinson, M. D. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx631 · Cited by 60 · see citing works
Self-citationQuantitative single-cell transcriptomics
Ziegenhain, C., Vieth, B., Parekh, S., Hellmann, I., & Enard, W. (2018) · Briefings in Functional Genomics · doi:10.1093/bfgp/ely009 · Cited by 58 · see citing works
ascend : R package for analysis of single-cell RNA-seq data
Senabouth, A., Lukowski, S. W., Alquicira-Hernández, J., Andersen, S. B., Mei, X., Nguyen, Q. H., & Powell, J. E. (2019) · GigaScience · doi:10.1093/gigascience/giz087 · Cited by 46 · see citing works
BEARscc determines robustness of single-cell clusters using simulated technical replicates
Severson, D. T., Owen, R. P., White, M. J., Lü, X., & Schuster‐Böckler, B. (2018) · Nature Communications · doi:10.1038/s41467-018-03608-y · Cited by 16 · see citing works
ascend : R package for analysis of single cell RNA-seq dataPreprint
Senabouth, A., Lukowski, S. W., Alquicira-Hernández, J., et al. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/207704 · Cited by 15 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Current best practices in single-cell RNA-seq analysis: a tutorial
Luecken, M. D., & Theis, F. J. (2019) · Molecular Systems Biology · doi:10.15252/msb.20188746 · Cited by 2,231 · see citing works
Eleven grand challenges in single-cell data science
Lähnemann, D., Köster, J., Szczurek, E., McCarthy, D. J., Hicks, S. C., Robinson, M. D., Vallejos, C. A., Campbell, K. R., Beerenwinkel, N., Mahfouz, A., et al. (2020) · Genome Biology · doi:10.1186/s13059-020-1926-6 · Cited by 1,327 · see citing works
Self-citationBest practices for single-cell analysis across modalities
Heumos, L., Schaar, A. C., Lance, C., Litinetskaya, A., Drost, F., Zappia, L., Lücken, M. D., Strobl, D. C., Henao, J., Curion, F., et al. (2023) · Nature Reviews Genetics · doi:10.1038/s41576-023-00586-w · Cited by 1,051 · see citing works
Integrating single-cell and spatial transcriptomics to elucidate intercellular tissue dynamics
Longo, S. K., Guo, M. G., Ji, A. L., & Khavari, P. A. (2021) · Nature Reviews Genetics · doi:10.1038/s41576-021-00370-8 · Cited by 1,059 · see citing works
Orchestrating single-cell analysis with Bioconductor
Amezquita, R. A., Lun, A. T. L., Becht, E., Carey, V. J., Carpp, L. N., Geistlinger, L., Marini, F., Rue-Albrecht, K., Risso, D., Soneson, C., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0654-x · Cited by 945 · see citing works
Self-citationA comparison of automatic cell identification methods for single-cell RNA sequencing data
Abdelaal, T., Michielsen, L., Cats, D., Hoogduin, D., Mei, H., Reinders, M. J. T., & Mahfouz, A. (2019) · Genome Biology · doi:10.1186/s13059-019-1795-z · Cited by 628 · see citing works
Crowell, H. L., Soneson, C., Germain, P.-L., Calini, D., Collin, L., Raposo, C., Malhotra, D., & Robinson, M. D. (2020) · Nature Communications · doi:10.1038/s41467-020-19894-4 · Cited by 515 · see citing works
Self-citationSingle-cell multiomics: technologies and data analysis methods
Lee, J., Hyeon, D. Y., & Hwang, D. (2020) · Experimental & Molecular Medicine · doi:10.1038/s12276-020-0420-2 · Cited by 524 · see citing works
Feature selection and dimension reduction for single-cell RNA-Seq based on a multinomial model
Townes, F. W., Hicks, S. C., Aryee, M. J., & Irizarry, R. A. (2019) · Genome Biology · doi:10.1186/s13059-019-1861-6 · Cited by 471 · see citing works
Benchmarking Computational Doublet-Detection Methods for Single-Cell RNA Sequencing Data
Xi, N. M., & Li, J. J. (2021) · Cell Systems · doi:10.1016/j.cels.2020.11.008 · Cited by 208 · see citing works
Sun, S., Zhu, J., Ma, Y., & Zhou, X. (2019) · Genome biology · doi:10.1186/s13059-019-1898-6 · Cited by 225 · see citing works
Analytic Pearson residuals for normalization of single-cell RNA-seq UMI data
Lause, J., Berens, P., & Kobak, D. (2021) · Genome Biology · doi:10.1186/s13059-021-02451-7 · Cited by 195 · see citing works
clustifyr: an R package for automated single-cell RNA sequencing cluster classification
Fu, R., Gillen, A. E., Sheridan, R. M., Tian, C., Daya, M., Hao, Y., Hesselberth, J. R., & Riemondy, K. A. (2020) · F1000Research · doi:10.12688/f1000research.22969.2 · Cited by 135 · see citing works
Spearheading future omics analyses using dyngen, a multi-modal simulator of single cells
Cannoodt, R., Saelens, W., Deconinck, L., & Saeys, Y. (2021) · Nature Communications · doi:10.1038/s41467-021-24152-2 · Cited by 124 · see citing works
Automatic cell type identification methods for single-cell RNA sequencing
Xie, B., Jiang, Q., Mora, A., & Li, X. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.10.027 · Cited by 66 · see citing works
- Authors
- Duò, A., Robinson, M. D., & Soneson, C.
- Affiliations
- SIB Swiss Institute of Bioinformatics (CH)
- Corresponding author
- Soneson, Charlotte
- Venue
- F1000Research
- Publisher
- F1000 Research Ltd
- Published
- 2018-09-10
- Volume
- 7
- Pages
- 1141
- ISSN
- 2046-1402 (ISSN-L)
- PMC
- PMC6134335
- OpenAlex
- W2884197618
- Cited by
- 296
- References
- 60
- Citations by year
- 2026: 8 · 2025: 19 · 2024: 27 · 2023: 39 · 2022: 46 · 2021: 65 · 2020: 55 · 2019: 35 · 2018: 2
- FWCI
- 13.96
- Type
- journal article
- Language
- en
- Update notices
- new_version (2020-11-16)
- Open access
- gold
- doi.org
- Locations
- F1000Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
- Zurich Open Repository and Archive (University of Zurich)
- Funders
- Chan Zuckerberg Initiative (182828)
- Swiss National Science Foundation (310030_175841)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Clustering Algorithms Research
- Keywords
- single-cell RNA sequencing, unsupervised clustering, clustering algorithm comparison, subpopulation identification, consensus clustering, Sc3+, Seurat, clustering stability, computational scalability, feature selection, simulated data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
