The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Comparison of clustering methods for high‐dimensional single‐cell flow and mass cytometry data
Open accessAbstract
Recent technological developments in high‐dimensional flow cytometry and mass cytometry (CyTOF) have made it possible to detect expression levels of dozens of protein markers in thousands of cells per second, allowing cell populations to be characterized in unprecedented detail. Traditional data analysis by “manual gating” can be inefficient and unreliable in these high‐dimensional settings, which has led to the development of a large number of automated analysis methods. Methods designed for unsupervised analysis use specialized clustering algorithms to detect and define cell populations for further downstream analysis. Here, we have performed an up‐to‐date, extensible performance comparison of clustering methods for high‐dimensional flow and mass cytometry data. We evaluated methods using several publicly available data sets from experiments in immunology, containing both major and rare cell populations, with cell population identities from expert manual gating as the reference standard. Several methods performed well, including FlowSOM, X‐shift, PhenoGraph, Rclusterpp, and flowMeans. Among these, FlowSOM had extremely fast runtimes, making this method well‐suited for interactive, exploratory analysis of large, high‐dimensional data sets on a standard laptop or desktop computer. These results extend previously published comparisons by focusing on high‐dimensional data and including new methods developed for CyTOF data. R scripts to reproduce all analyses are available from GitHub ( https://github.com/lmweber/cytometry-clustering-comparison ), and pre‐processed data files are available from FlowRepository (FR‐FCM‐ZZPH), allowing our comparisons to be extended to include new clustering methods and reference data sets. © 2016 The Authors. Cytometry Part A published by Wiley Periodicals, Inc. on behalf of ISAC.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Orchestrating high-throughput genomic analysis with Bioconductor
Huber, W., Carey, V. J., Gentleman, R., Anders, S., Carlson, M., Carvalho, B. S., Bravo, H. C., Davis, S., Gatto, L., Girke, T., et al. (2015) · Nature Methods · doi:10.1038/nmeth.3252 · Cited by 3,608
Hematopoiesis: An Evolving Paradigm for Stem Cell Biology
Orkin, S. H., & Zon, L. I. (2008) · Cell · doi:10.1016/j.cell.2008.01.025 · Cited by 2,570
Data-Driven Phenotypic Dissection of AML Reveals Progenitor-like Cells that Correlate with Prognosis
Levine, J. H., Simonds, E. F., Bendall, S. C., Davis, K. L., Amir, E.-A. D., Tadmor, M. D., Litvin, O., Fienberg, H. G., Jager, A., Zunder, E. R., et al. (2015) · Cell · doi:10.1016/j.cell.2015.05.047 · Cited by 2,303
Bendall, S. C., Simonds, E. F., Qiu, P., Amir, E.-A. D., Krutzik, P. O., Finck, R., Bruggner, R. V., Melamed, R., Trejo, A., Ornatsky, O. I., et al. (2011) · Science · doi:10.1126/science.1198704 · Cited by 2,279
FlowSOM: Using self‐organizing maps for visualization and interpretation of cytometry data
Van Gassen, S., Callebaut, B., Van Helden, M. J., Lambrecht, B. N., Demeester, P., Dhaene, T., & Saeys, Y. (2015) · Cytometry Part A · doi:10.1002/cyto.a.22625 · Cited by 1,919
Kriegel, H.-P., Kröger, P., & Zimek, A. (2009) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/1497577.1497578 · Cited by 1,093
Extracting a cellular hierarchy from high-dimensional cytometry data with SPADE
Qiu, P., Simonds, E. F., Bendall, S. C., Gibbs, K. D., Bruggner, R. V., Linderman, M. D., Sachs, K., Nolan, G. P., & Plevritis, S. K. (2011) · Nature Biotechnology · doi:10.1038/nbt.1991 · Cited by 956
A deep profiler's guide to cytometry
Bendall, S. C., Nolan, G. P., Roederer, M., & Chattopadhyay, P. K. (2012) · Trends in Immunology · doi:10.1016/j.it.2012.02.010 · Cited by 656
Critical assessment of automated flow cytometry data analysis techniques
The FlowCAP Consortium, Aghaeepour, N., The DREAM Consortium, Finak, G., Hoos, H., Mosmann, T. R., Brinkman, R., Gottardo, R., & Scheuermann, R. H. (2013) · Nature Methods · doi:10.1038/nmeth.2365 · Cited by 598
Computational flow cytometry: helping to make sense of high-dimensional immunology data
Saeys, Y., Van Gassen, S., & Lambrecht, B. N. (2016) · Nature Reviews Immunology · doi:10.1038/nri.2016.56 · Cited by 467
Cited by
Current best practices in single-cell RNA-seq analysis: a tutorial
Luecken, M. D., & Theis, F. J. (2019) · Molecular Systems Biology · doi:10.15252/msb.20188746 · Cited by 2,231
Eleven grand challenges in single-cell data science
Lähnemann, D., Köster, J., Szczurek, E., McCarthy, D. J., Hicks, S. C., Robinson, M. D., Vallejos, C. A., Campbell, K. R., Beerenwinkel, N., Mahfouz, A., et al. (2020) · Genome Biology · doi:10.1186/s13059-020-1926-6 · Cited by 1,327
Self-citationGuidelines for the use of flow cytometry and cell sorting in immunological studies (second edition)
Cossarizza, A., Chang, H., Radbruch, A., Acs, A., Adam, D., Adam‐Klages, S., Agace, W. W., Aghaeepour, N., Akdis, M., Allez, M., et al. (2019) · European Journal of Immunology · doi:10.1002/eji.201970107 · Cited by 979
A Single-Cell Atlas of the Tumor and Immune Ecosystem of Human Breast Cancer
Wagner, J., Rapsomaniki, M. A., Chevrier, S., Anzeneder, T., Langwieder, C., Dykgers, A., Rees, M., Ramaswamy, A., Muenst, S., Soysal, S. D., et al. (2019) · Cell · doi:10.1016/j.cell.2019.03.005 · Cited by 790
High-dimensional single-cell analysis predicts response to anti-PD-1 immunotherapy
Krieg, C., Nowicka, M., Guglietta, S., Schindler, S., Hartmann, F. J., Weber, L. M., Dummer, R., Robinson, M. D., Levesque, M. P., & Becher, B. (2018) · Nature Medicine · doi:10.1038/nm.4466 · Cited by 735
Self-citationStereotypic Immune System Development in Newborn Children
Olin, A., Henckel, E., Chen, Y., Lakshmikanth, T., Pou, C., Mikes, J., Gustafsson, A., Bernhardsson, A. K., Zhang, C., Bohlin, K., et al. (2018) · Cell · doi:10.1016/j.cell.2018.06.045 · Cited by 696
histoCAT: analysis of cell phenotypes and interactions in multiplex image cytometry data
Schapiro, D., Jackson, H. W., Raghuraman, S., Fischer, J. R., Zanotelli, V. R. T., Schulz, D., Giesen, C., Catena, R., Varga, Z., & Bodenmiller, B. (2017) · Nature Methods · doi:10.1038/nmeth.4391 · Cited by 659
Belkina, A. C., Ciccolella, C. O., Anno, R., Halpert, R., Spidlen, J., & Snyder-Cappione, J. E. (2019) · Nature Communications · doi:10.1038/s41467-019-13055-y · Cited by 625
Guidelines for the use of flow cytometry and cell sorting in immunological studies *
Cossarizza, A., Chang, H., Radbruch, A., Akdis, M., Andrä, I., Annunziato, F., Bacher, P., Barnaba, V., Battistini, L., Bauer, W. M., et al. (2017) · European Journal of Immunology · doi:10.1002/eji.201646632 · Cited by 542
Multiplexed protein maps link subcellular organization to cellular states
Gut, G., Herrmann, M. D., & Pelkmans, L. (2018) · Science · doi:10.1126/science.aar7042 · Cited by 510
CyTOF workflow: Differential discovery in high-throughput high-dimensional cytometry datasets
Nowicka, M., Krieg, C., Weber, L. M., Hartmann, F. J., Guglietta, S., Becher, B., Levesque, M. P., & Robinson, M. D. (2017) · F1000Research · doi:10.12688/f1000research.11622.1 · Cited by 403
Self-citationCyTOF workflow: differential discovery in high-throughput high-dimensional cytometry datasets
Nowicka, M., Krieg, C., Crowell, H. L., Weber, L. M., Hartmann, F. J., Guglietta, S., Becher, B., Levesque, M. P., & Robinson, M. D. (2019) · F1000Research · doi:10.12688/f1000research.11622.3 · Cited by 399
Self-citationA systematic performance evaluation of clustering methods for single-cell RNA-seq data
Duò, A., Robinson, M. D., & Soneson, C. (2018) · F1000Research · doi:10.12688/f1000research.15666.2 · Cited by 296
Self-citationdiffcyt: Differential discovery in high-dimensional cytometry via high-resolution clustering
Weber, L. M., Nowicka, M., Soneson, C., & Robinson, M. D. (2019) · Communications Biology · doi:10.1038/s42003-019-0415-5 · Cited by 273
Self-citationCytoNorm: A Normalization Algorithm for Cytometry Data
Van Gassen, S., Gaudilliere, B., Angst, M. S., Saeys, Y., & Aghaeepour, N. (2020) · Cytometry Part A · doi:10.1002/cyto.a.23904 · Cited by 234
Sun, S., Zhu, J., Ma, Y., & Zhou, X. (2019) · Genome biology · doi:10.1186/s13059-019-1898-6 · Cited by 225
Comprehensive Immune Monitoring of Clinical Trials to Advance Human Immunotherapy
Hartmann, F. J., Babdor, J., Gherardini, P. F., Amir, E.-A. D., Jones, K., Sahaf, B., Marquez, D. M., Krutzik, P., O’Donnell, E., Sigal, N., et al. (2019) · Cell Reports · doi:10.1016/j.celrep.2019.06.049 · Cited by 124
Optimal transport for single-cell and spatial omics
Bunne, C., Schiebinger, G., Krause, A., Regev, A., & Cuturi, M. (2024) · Nature Reviews Methods Primers · doi:10.1038/s43586-024-00334-2 · Cited by 71
Rasouli, J., Casella, G., Yoshimura, S., Zhang, W., Xiao, D., Garifallou, J., Gonzalez, M. V., Wiedeman, A., Kus, A., Mari, E. R., et al. (2020) · Science Immunology · doi:10.1126/sciimmunol.aba9953 · Cited by 57
Removing unwanted variation with CytofRUV to integrate multiple CyTOF datasets
Trussart, M., Teh, C. E., Tan, T., Leong, L., Gray, D. H., & Speed, T. P. (2020) · eLife · doi:10.7554/elife.59630 · Cited by 51
Ivison, S., Malek, M., Garcia, R. V., Broady, R., Halpin, A., Richaud, M., Brant, R. F., Wang, S.-I., Goupil, M., Guan, Q., et al. (2018) · JCI Insight · doi:10.1172/jci.insight.121867 · Cited by 49
Essential guidelines for computational method benchmarking
Weber, L. M., Saelens, W., Cannoodt, R., Soneson, C., Hapfelmeier, A., Gardner, P. P., Boulesteix, A.-L., Saeys, Y., & Robinson, M. D. (2019) · Genome Biology · doi:10.1186/s13059-019-1738-8 · Cited by 21
Self-citation
Metadata
- Authors
- Weber, L. M., & Robinson, M. D.
- Venue
- Cytometry Part A
- Publisher
- Wiley
- Published
- 2016-12
- Volume
- 89
- Issue
- 12
- Pages
- 1084-1096
- ISSN
- 1552-4922 (ISSN-L)
- 1552-4930
- OpenAlex
- W2951566836
- Cited by
- 378
- References
- 38
- FWCI
- 18.06
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.