The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Comparison of clustering methods for high‐dimensional single‐cell flow and mass cytometry data
Open accessAbstract
Recent technological developments in high‐dimensional flow cytometry and mass cytometry (CyTOF) have made it possible to detect expression levels of dozens of protein markers in thousands of cells per second, allowing cell populations to be characterized in unprecedented detail. Traditional data analysis by “manual gating” can be inefficient and unreliable in these high‐dimensional settings, which has led to the development of a large number of automated analysis methods. Methods designed for unsupervised analysis use specialized clustering algorithms to detect and define cell populations for further downstream analysis. Here, we have performed an up‐to‐date, extensible performance comparison of clustering methods for high‐dimensional flow and mass cytometry data. We evaluated methods using several publicly available data sets from experiments in immunology, containing both major and rare cell populations, with cell population identities from expert manual gating as the reference standard. Several methods performed well, including FlowSOM, X‐shift, PhenoGraph, Rclusterpp, and flowMeans. Among these, FlowSOM had extremely fast runtimes, making this method well‐suited for interactive, exploratory analysis of large, high‐dimensional data sets on a standard laptop or desktop computer. These results extend previously published comparisons by focusing on high‐dimensional data and including new methods developed for CyTOF data. R scripts to reproduce all analyses are available from GitHub ( https://github.com/lmweber/cytometry-clustering-comparison ), and pre‐processed data files are available from FlowRepository (FR‐FCM‐ZZPH), allowing our comparisons to be extended to include new clustering methods and reference data sets. © 2016 The Authors. Cytometry Part A published by Wiley Periodicals, Inc. on behalf of ISAC.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Orchestrating high-throughput genomic analysis with Bioconductor
Huber, W., Carey, V. J., Gentleman, R., Anders, S., Carlson, M., Carvalho, B. S., Bravo, H. C., Davis, S., Gatto, L., Girke, T., et al. (2015) · Nature Methods · doi:10.1038/nmeth.3252 · Cited by 3,608
Hematopoiesis: An Evolving Paradigm for Stem Cell Biology
Orkin, S. H., & Zon, L. I. (2008) · Cell · doi:10.1016/j.cell.2008.01.025 · Cited by 2,570
Data-Driven Phenotypic Dissection of AML Reveals Progenitor-like Cells that Correlate with Prognosis
Levine, J. H., Simonds, E. F., Bendall, S. C., Davis, K. L., Amir, E.-A. D., Tadmor, M. D., Litvin, O., Fienberg, H. G., Jager, A., Zunder, E. R., et al. (2015) · Cell · doi:10.1016/j.cell.2015.05.047 · Cited by 2,303
Bendall, S. C., Simonds, E. F., Qiu, P., Amir, E.-A. D., Krutzik, P. O., Finck, R., Bruggner, R. V., Melamed, R., Trejo, A., Ornatsky, O. I., et al. (2011) · Science · doi:10.1126/science.1198704 · Cited by 2,279
FlowSOM: Using self‐organizing maps for visualization and interpretation of cytometry data
Van Gassen, S., Callebaut, B., Van Helden, M. J., Lambrecht, B. N., Demeester, P., Dhaene, T., & Saeys, Y. (2015) · Cytometry Part A · doi:10.1002/cyto.a.22625 · Cited by 1,919
Kriegel, H.-P., Kröger, P., & Zimek, A. (2009) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/1497577.1497578 · Cited by 1,093
Extracting a cellular hierarchy from high-dimensional cytometry data with SPADE
Qiu, P., Simonds, E. F., Bendall, S. C., Gibbs, K. D., Bruggner, R. V., Linderman, M. D., Sachs, K., Nolan, G. P., & Plevritis, S. K. (2011) · Nature Biotechnology · doi:10.1038/nbt.1991 · Cited by 956
A deep profiler's guide to cytometry
Bendall, S. C., Nolan, G. P., Roederer, M., & Chattopadhyay, P. K. (2012) · Trends in Immunology · doi:10.1016/j.it.2012.02.010 · Cited by 656
Critical assessment of automated flow cytometry data analysis techniques
The FlowCAP Consortium, Aghaeepour, N., The DREAM Consortium, Finak, G., Hoos, H., Mosmann, T. R., Brinkman, R., Gottardo, R., & Scheuermann, R. H. (2013) · Nature Methods · doi:10.1038/nmeth.2365 · Cited by 598
Computational flow cytometry: helping to make sense of high-dimensional immunology data
Saeys, Y., Van Gassen, S., & Lambrecht, B. N. (2016) · Nature Reviews Immunology · doi:10.1038/nri.2016.56 · Cited by 467
Automated mapping of phenotype space with single-cell data
Samusik, N., Good, Z., Spitzer, M. H., Davis, K. L., & Nolan, G. P. (2016) · Nature Methods · doi:10.1038/nmeth.3863 · Cited by 419
High-dimensional analysis of the murine myeloid cell system
Becher, B., Schlitzer, A., Chen, J., Mair, F., Sumatoh, H. R., Teng, K. W. W., Low, D., Ruedl, C., Riccardi-Castagnoli, P., Poidinger, M., et al. (2014) · Nature Immunology · doi:10.1038/ni.3006 · Cited by 357
Cytofkit: A Bioconductor Package for an Integrated Mass Cytometry Data Analysis Pipeline
Chen, H., Lau, M. C., Wong, M. T., Newell, E. W., Poidinger, M., & Chen, J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005112 · Cited by 355
Standardizing Flow Cytometry Immunophenotyping Analysis from the Human ImmunoPhenotyping Consortium
Finak, G., Langweiler, M., Jaimes, M., Malek, M., Taghiyar, J., Korin, Y., Raddassi, K., Devine, L., Obermoser, G., Pekalski, M. L., et al. (2016) · Scientific Reports · doi:10.1038/srep20686 · Cited by 285
Hornik, K. (2005) · Journal of Statistical Software · doi:10.18637/jss.v014.i12 · Cited by 268
The end of gating? An introduction to automated analysis of high dimensional cytometry data
Mair, F., Hartmann, F. J., Mrdjen, D., Tosevski, V., Krieg, C., & Becher, B. (2016) · European Journal of Immunology · doi:10.1002/eji.201545774 · Cited by 234
Comparing the performance of biomedical clustering methods
Wiwie, C., Baumbach, J., & Röttger, R. (2015) · Nature Methods · doi:10.1038/nmeth.3583 · Cited by 224
Rapid cell population identification in flow cytometry data
Aghaeepour, N., Nikolic, R., Hoos, H. H., & Brinkman, R. R. (2011) · Cytometry Part A · doi:10.1002/cyto.a.21007 · Cited by 221
Automatic Classification of Cellular Expression by Nonlinear Stochastic Embedding (ACCENSE)
Shekhar, K., Brodin, P., Davis, M. M., & Chakraborty, A. K. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1321405111 · Cited by 215
Finak, G., Frelinger, J., Jiang, W., Newell, E. W., Ramey, J., Davis, M. M., Kalams, S. A., De Rosa, S. C., & Gottardo, R. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003806 · Cited by 197
Qian, Y., Wei, C., Eun‐Hyung Lee, F., Campbell, J., Halliley, J., Lee, J. A., Cai, J., Kong, Y. M., Sadat, E., Thomson, E., et al. (2010) · Cytometry Part B: Clinical Cytometry · doi:10.1002/cyto.b.20554 · Cited by 192
Spidlen, J., Breuer, K., Rosenberg, C., Kotecha, N., & Brinkman, R. R. (2012) · Cytometry Part A · doi:10.1002/cyto.a.22106 · Cited by 186
flowClust: a Bioconductor package for automated gating of flow cytometry data
Lo, K., Hahne, F., Brinkman, R. R., & Gottardo, R. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-145 · Cited by 185
Avoiding common pitfalls when clustering biological data
Ronan, T., Qi, Z., & Naegle, K. M. (2016) · Science Signaling · doi:10.1126/scisignal.aad1932 · Cited by 174
Data reduction for spectral clustering to analyze high throughput flow cytometry data
Zare, H., Shooshtari, P., Gupta, A., & Brinkman, R. R. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-403 · Cited by 161
Mass cytometry: blessed with the curse of dimensionality
Newell, E. W., & Cheng, Y. (2016) · Nature Immunology · doi:10.1038/ni.3485 · Cited by 144
Methods for discovery and characterization of cell subsets in high dimensional mass cytometry data
Diggins, K. E., Ferrell, P. B., & Irish, J. M. (2015) · Methods · doi:10.1016/j.ymeth.2015.05.008 · Cited by 143
Ge, Y., & Sealfon, S. C. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts300 · Cited by 135
van Unen, V., Li, N., Molendijk, I., Temurhan, M., Höllt, T., Jong, A. E. V. D. M., Verspaget, H. W., Mearin, M. L., Mulder, C. J. J., van Bergen, J., et al. (2016) · Immunity · doi:10.1016/j.immuni.2016.04.014 · Cited by 128
High-dimensional single-cell analysis reveals the immune signature of narcolepsy
Hartmann, F. J., Bernard‐Valnet, R., Queriault, C., Mrdjen, D., Weber, L. M., Galli, E., Krieg, C., Robinson, M. D., Nguyen, X., Dauvilliers, Y. A., et al. (2016) · The Journal of Experimental Medicine · doi:10.1084/jem.20160897 · Cited by 126
Self-citationMerging Mixture Components for Cell Population Identification in Flow Cytometry
Finak, G., Bashashati, A., Brinkman, R. R., & Gottardo, R. (2009) · Advances in Bioinformatics · doi:10.1155/2009/247646 · Cited by 102
Algorithmic Tools for Mining High-Dimensional Cytometry Data
Chester, C., & Maecker, H. T. (2015) · The Journal of Immunology · doi:10.4049/jimmunol.1500633 · Cited by 99
Mosmann, T. R., Naim, I., Rebhahn, J. A., Datta, S., Cavenaugh, J. S., Weaver, J. M., & Gaurav Sharma, A. (2014) · Cytometry Part A · doi:10.1002/cyto.a.22445 · Cited by 88
Aghaeepour, N., Chattopadhyay, P. K., Chikina, M. D., Dhaene, T., Gassen, S. V., Kursa, M. B., Lambrecht, B. N., Malek, M., McLachlan, G. J., Yu, Q., et al. (2015) · Cytometry Part A · doi:10.1002/cyto.a.22732 · Cited by 69
Sörensen, T., Baumgart, S., Durek, P., Grützkau, A., & Häupl, T. (2015) · Cytometry Part A · doi:10.1002/cyto.a.22626 · Cited by 58
Still Alive and Kicking: In-Vitro-Generated GM-CSF Dendritic Cells!
Lutz, M. B., Inaba, K., Schuler, G., & Romani, N. (2016) · Immunity · doi:10.1016/j.immuni.2015.12.013 · Cited by 58
Frequency determination of rare populations by flow cytometry: A hematopoietic stem cell perspective
Nilsson, A. R., Bryder, D., & Pronk, C. J. (2013) · Cytometry Part A · doi:10.1002/cyto.a.22324 · Cited by 44
Pejoski, D., Tchitchek, N., Pozo, A. R., Elhmouzi‐Younes, J., Yousfi, R., Rogez‐Kreuz, C., Clayette, P., Dereuddre‐Bosquet, N., Lévy, Y., Cosma, A., et al. (2016) · The Journal of Immunology · doi:10.4049/jimmunol.1502005 · Cited by 29
Cited by
Current best practices in single-cell RNA-seq analysis: a tutorial
Luecken, M. D., & Theis, F. J. (2019) · Molecular Systems Biology · doi:10.15252/msb.20188746 · Cited by 2,231
Eleven grand challenges in single-cell data science
Lähnemann, D., Köster, J., Szczurek, E., McCarthy, D. J., Hicks, S. C., Robinson, M. D., Vallejos, C. A., Campbell, K. R., Beerenwinkel, N., Mahfouz, A., et al. (2020) · Genome Biology · doi:10.1186/s13059-020-1926-6 · Cited by 1,327
Self-citationGuidelines for the use of flow cytometry and cell sorting in immunological studies (second edition)
Cossarizza, A., Chang, H., Radbruch, A., Acs, A., Adam, D., Adam‐Klages, S., Agace, W. W., Aghaeepour, N., Akdis, M., Allez, M., et al. (2019) · European Journal of Immunology · doi:10.1002/eji.201970107 · Cited by 979
A Single-Cell Atlas of the Tumor and Immune Ecosystem of Human Breast Cancer
Wagner, J., Rapsomaniki, M. A., Chevrier, S., Anzeneder, T., Langwieder, C., Dykgers, A., Rees, M., Ramaswamy, A., Muenst, S., Soysal, S. D., et al. (2019) · Cell · doi:10.1016/j.cell.2019.03.005 · Cited by 790
High-dimensional single-cell analysis predicts response to anti-PD-1 immunotherapy
Krieg, C., Nowicka, M., Guglietta, S., Schindler, S., Hartmann, F. J., Weber, L. M., Dummer, R., Robinson, M. D., Levesque, M. P., & Becher, B. (2018) · Nature Medicine · doi:10.1038/nm.4466 · Cited by 735
Self-citation
Metadata
- Authors
- Weber, L. M., & Robinson, M. D.
- Venue
- Cytometry Part A
- Publisher
- Wiley
- Published
- 2016-12
- Volume
- 89
- Issue
- 12
- Pages
- 1084-1096
- ISSN
- 1552-4922 (ISSN-L)
- 1552-4930
- OpenAlex
- W2951566836
- Cited by
- 378
- References
- 38
- FWCI
- 18.06
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.