CoChat

A systematic performance evaluation of clustering methods for single-cell RNA-seq data

Open access
Duò, Angelo, Robinson, Mark D., Soneson, Charlotte · 2018 · F1000Research
Cited by 93

The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.

Abstract

Subpopulation identification, usually via some form of unsupervised clustering, is a fundamental step in the analysis of many single-cell RNA-seq data sets. This has motivated the development and application of a broad range of clustering methods, based on various underlying algorithms. Here, we provide a systematic and extensible performance evaluation of 12 clustering algorithms, including both methods developed explicitly for scRNA-seq data and more general-purpose methods. The methods were evaluated using 9 publicly available scRNA-seq data sets as well as three simulations with varying degree of cluster separability. The same feature selection approaches were used for all methods, allowing us to focus on the investigation of the performance of the clustering algorithms themselves. We evaluated the ability of recovering known subpopulations, the stability and the run time of the methods. Additionally, we investigated whether the performance could be improved by generating consensus partitions from multiple individual clustering methods. We found substantial differences in the performance, run time and stability between the methods, with SC3 and Seurat showing the most favorable results. Additionally, we found that consensus clustering typically did not improve the performance compared to the best of the combined methods, but that several of the top-performing methods already perform some type of consensus clustering. The R scripts providing an extensible framework for the evaluation of new methods and data sets are available on GitHub ( https://github.com/markrobinsonuzh/scRNAseq_clustering_comparison ).

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.12688/f1000research.15666.1

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 23 references

Research works printed but not linked: 30

  1. E Macosko, 2015, Highly Parallel Genome-wide Expression Profiling of Individual Cells Using Nanoliter Droplets., Cell., 161, 1202-1214, 10.1016/j.cell.2015.05.002 · DOI · Paper page
  2. V Svensson, 2018, Exponential scaling of single-cell RNA-seq in the past decade., Nat Protoc., 13, 599-604, 10.1038/nprot.2017.149 · DOI · Paper page
  3. C Ziegenhain, 2018, Quantitative single-cell transcriptomics., Brief Funct Genomics., ely009, 10.1093/bfgp/ely009 · DOI
  4. R Bacher, 2016, Design and computational analysis of single-cell RNA-sequencing experiments., Genome Biol., 17, 63, 10.1186/s13059-016-0927-y · DOI · Paper page
  5. N Aghaeepour, 2013, Critical assessment of automated flow cytometry data analysis techniques., Nat Methods., 10, 228-238, 10.1038/nmeth.2365 · DOI · Paper page
  6. L Weber, 2016, Comparison of clustering methods for high-dimensional single-cell flow and mass cytometry data., Cytometry A., 89, 1084-1096, 10.1002/cyto.a.23030 · DOI · Paper page
  7. V Menon, 2017, Clustering single cells: a review of approaches on high-and low-depth single-cell RNA-seq data., Brief Funct Genomics., elx044, 10.1093/bfgp/elx044 · DOI
  8. P Langfelder, 2008, WGCNA: an R package for weighted correlation network analysis., BMC Bioinformatics., 9, 559, 10.1186/1471-2105-9-559 · DOI · Paper page
  9. S Freytag, 2017, Cluster headache: Comparing clustering tools for 10X single cell sequencing data., bioRxiv., 10.1101/203752 · DOI
  10. T Andrews, 2018, Identifying cell populations with scRNASeq., Mol Aspects Med., 59, 114-122, 10.1016/j.mam.2017.07.002 · DOI · Paper page
  11. P Koh, 2016, An atlas of transcriptional, chromatin accessibility, and surface marker changes in human mesoderm development., Sci Data., 3, 10.1038/sdata.2016.109 · DOI
  12. M Ramos, 2017, Software for the integration of Multi-Omics experiments in Bioconductor., bioRxiv., 10.1101/144774 · DOI
  13. V Ntranos, 2016, Fast and accurate single-cell RNA-Seq analysis by clustering of transcript-compatibility counts., Genome Biol., 17, 112, 10.1186/s13059-016-0970-8 · DOI
  14. C Soneson, 2018, Towards unified quality verification of synthetic count data with countsimQC., Bioinformatics., 34, 691-692, 10.1093/bioinformatics/btx631 · DOI · Paper page
  15. K Pearson, 1901, On lines and planes of closest fit to systems of points in space., Philos Mag., 2, 559-572, 10.1080/14786440109462720 · DOI · Paper page
  16. L van der Maaten, 2008, Visualizing data using t-SNE., J Mach Learn Res., 9, 2579-2605
  17. T Andrews, 2018, Dropout-based feature selection for scRNASeq., bioRxiv., 10.1101/065094 · DOI
  18. A Senabouth, 2017, ascend: R package for analysis of single cell RNA-seq data., bioRxiv., 10.1101/207704 · DOI
  19. P Lin, 2017, CIDR: Ultrafast and accurate clustering through imputation for single-cell RNA-seq data., Genome Biol., 18, 59, 10.1186/s13059-017-1188-0 · DOI · Paper page
  20. S Van Gassen, 2015, Flowsom: Using self-organizing maps for visualization and interpretation of cytometry data., Cytometry A., 87, 636-645, 10.1002/cyto.a.22625 · DOI · Paper page
  21. J Ward, 1963, Hierarchical grouping to optimize an objective function., J Am Stat Assoc., 58, 236-244, 10.1080/01621459.1963.10500845 · DOI · Paper page
  22. J Žurauskienė, 2016, pcaReduce: hierarchical clustering of single cell transcriptional profiles., BMC Bioinformatics., 17, 140, 10.1186/s12859-016-0984-y · DOI · Paper page
  23. L Van Der Maaten, 2014, Accelerating t-SNE using tree-based algorithms., J Mach Learn Res., 15, 1-21
  24. Y Yang, 2017, SAFE-clustering: Single-cell aggregated (from ensemble) clustering for single-cell RNA-seq data., bioRxiv., 10.1101/215723 · DOI
  25. Z Ji, 2016, TSCAN: Pseudo-time reconstruction and evaluation in single-cell RNA-seq analysis., Nucleic Acids Res., 44, e117, 10.1093/nar/gkw430 · DOI · Paper page
  26. C Shannon, 1948, A mathematical theory of communication., Bell Syst Tech J., 27, 379-423, 10.1002/j.1538-7305.1948.tb01338.x · DOI · Paper page
  27. K Hornik, 2005, A CLUE for CLUster Ensembles., J Stat Softw., 14, 1-25, 10.18637/jss.v014.i12 · DOI · Paper page
  28. U Von Luxburg, 2010, Clustering stability: an overview., Foundations and Trends in Machine Learning., 2, 235-274, 10.1561/2200000008 · DOI · Paper page
  29. D Severson, 2018, BEARscc determines robustness of single-cell clusters using simulated technical replicates., Nat Commun., 9, 10.1038/s41467-018-03608-y · DOI
  30. A Duò, 2018, markrobinsonuzh/scRNAseq_clustering_comparison: F1000 v1 (Version 0.9)., Zenodo.

Cited by

See all 93

Metadata

Authors
Duò, A., Robinson, M. D., & Soneson, C.
Venue
F1000Research
Published
2018-07-26
Volume
7
Pages
1141
ISSN
2046-1402 (ISSN-L)
OpenAlex
W4238363316
Cited by
93
References
23
FWCI
4.71
Type
journal article
Language
en
Open access
gold
Funders
  • Chan Zuckerberg Initiative (182828)
  • Swiss National Science Foundation (310030_175841)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A systematic performance evaluation of clustering methods for single-cell RNA-seq data. https://app.cochat.ai/papers/10.12688/f1000research.15666.1
The paper
Duò, A., Robinson, M. D., & Soneson, C. (2018). A systematic performance evaluation of clustering methods for single-cell RNA-seq data. F1000Research. https://doi.org/10.12688/f1000research.15666.1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.