Evaluation of methods to assign cell type labels to cell clusters from single-cell RNA-sequencing data
Open accessPublished version of a preprint
The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Background: Identification of cell type subpopulations from complex cell mixtures using single-cell RNA-sequencing (scRNA-seq) data includes automated computational steps like data normalization, dimensionality reduction and cell clustering. However, assigning cell type labels to cell clusters is still conducted manually by most researchers, resulting in limited documentation, low reproducibility and uncontrolled vocabularies. Two bottlenecks to automating this task are the scarcity of reference cell type gene expression signatures and the fact that some dedicated methods are available only as web servers with limited cell type gene expression signatures. Methods: In this study, we benchmarked four methods (CIBERSORT, GSEA, GSVA, and ORA) for the task of assigning cell type labels to cell clusters from scRNA-seq data. We used scRNA-seq datasets from liver, peripheral blood mononuclear cells and retinal neurons for which reference cell type gene expression signatures were available. Results: Our results show that, in general, all four methods show a high performance in the task as evaluated by receiver operating characteristic curve analysis (average area under the curve (AUC) = 0.94, sd = 0.036), whereas precision-recall curve analyses show a wide variation depending on the method and dataset (average AUC = 0.53, sd = 0.24). Conclusions: CIBERSORT and GSVA were the top two performers. Additionally, GSVA was the fastest of the four methods and was more robust in cell type gene expression signature subsampling simulations. We provide an extensible framework to evaluate other methods and datasets at https://github.com/jdime/scRNAseq_cell_cluster_labeling .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee, S., Ebert, B. L., Gillette, M. A., Paulovich, A., Pomeroy, S. L., Golub, T. R., Lander, E. S., et al. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0506580102 · Cited by 53,625
Single cell RNA sequencing of human liver reveals distinct intrahepatic macrophage populations
MacParland, S. A., Liu, J. C., Ma, X.-Z., Innes, B. T., Bartczak, A. M., Gage, B. K., Manuel, J., Khuu, N., Echeverri, J., Linares, I., et al. (2018) · Nature Communications · doi:10.1038/s41467-018-06318-7 · Cited by 1,552
Self-citationComprehensive Classification of Retinal Bipolar Neurons by Single-Cell Transcriptomics
Shekhar, K., Lapan, S. W., Whitney, I. E., Tran, N. M., Macosko, E. Z., Kowalczyk, M., Adiconis, X., Levin, J. Z., Nemesh, J., Goldman, M., et al. (2016) · Cell · doi:10.1016/j.cell.2016.07.054 · Cited by 1,176
The Human Cell Atlas: from vision to reality
Rozenblatt-Rosen, O., Stubbington, M. J. T., Regev, A., & Teichmann, S. A. (2017) · Nature · doi:10.1038/550451a · Cited by 638
Bard, J., Rhee, S. Y., & Ashburner, M. (2005) · Genome biology · doi:10.1186/gb-2005-6-2-r21 · Cited by 416
Crow, M., Paul, A., Ballouz, S., Huang, Z. J., & Gillis, J. (2018) · Nature Communications · doi:10.1038/s41467-018-03282-0 · Cited by 365
A systematic performance evaluation of clustering methods for single-cell RNA-seq data
Duò, A., Robinson, M. D., & Soneson, C. (2018) · F1000Research · doi:10.12688/f1000research.15666.2 · Cited by 296
Comparison of clustering tools in R for medium-sized 10x Genomics single-cell RNA-sequencing data
Freytag, S., Tian, L., Lönnstedt, I., Ng, M., & Bahlo, M. (2018) · F1000Research · doi:10.12688/f1000research.15809.1 · Cited by 220
Cell type discovery and representation in the era of high-content single cell phenotyping
Bakken, T., Cowell, L., Aevermann, B. D., Novotny, M., Hodge, R., Miller, J. A., Lee, A., Chang, I., McCorrison, J., Pulendran, B., et al. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1977-1 · Cited by 55
Research works printed but not linked: 13
- A Alavi, 2018, A web server for comparative analysis of single-cell RNA-seq data., Nat Commun., 9, 4768, 10.1038/s41467-018-07165-2 · DOI · Paper page
- J Alquicira-Hernandez, 2018, scPred: scPred: Cell type prediction at single-cell resolution., bioRxiv., 10.1101/369538 · DOI
- M Ashburner, 2000, Gene ontology: tool for the unification of biology. The Gene Ontology Consortium., Nat Genet., 25, 25-29, 10.1038/75556 · DOI · Paper page
- A Butler, 2018, Integrating single-cell transcriptomic data across different conditions, technologies, and species., Nat Biotechnol., 36, 411-420, 10.1038/nbt.4096 · DOI · Paper page
- J Diaz-Mejia, 2019b, Supplementary code for "Evaluation of methods to assign cell type labels to cell clusters from single-cell RNA-sequencing data" (Diaz-Mejia JJ et al., 2019) (Version v1.0)., Zenodo.
- R Fisher, 1935, The Logic of Inductive Inference., J R Stat Soc., 98, 39-82, 10.2307/2342435 · DOI · Paper page
- J Goeman, 2007, Analyzing gene expression data in terms of gene sets: methodological issues., Bioinformatics., 23, 980-987, 10.1093/bioinformatics/btm051 · DOI · Paper page
- S Hänzelmann, 2013, GSVA: gene set variation analysis for microarray and RNA-seq data., BMC Bioinformatics., 14, 7, 10.1186/1471-2105-14-7 · DOI · Paper page
- B Innes, 2018, scClustViz – Single-cell RNAseq cluster assessment and visualization [version 1; referees: 2 approved with reservations]., F1000Res., 7, 1522, 10.12688/f1000research.16198.1 · DOI
- A Newman, 2015a, Robust enumeration of cell subsets from tissue expression profiles., LM22 signature.
- A Newman, 2015b, Robust enumeration of cell subsets from tissue expression profiles., Nat Methods., 12, 453-457, 10.1038/nmeth.3337 · DOI · Paper page
- K Shekhar, 2016b, Comprehensive Classification of Retinal Bipolar Neurons by Single-Cell Transcriptomics., Cell., 166, 1308-1323.e30, 10.1016/j.cell.2016.07.054 · DOI · Paper page
- G Zheng, 2017a, Massively parallel digital transcriptional profiling of single cells., Nat Commun., 8, 10.1038/ncomms14049 · DOI · Paper page
Cited by
A comparison of automatic cell identification methods for single-cell RNA sequencing data
Abdelaal, T., Michielsen, L., Cats, D., Hoogduin, D., Mei, H., Reinders, M. J. T., & Mahfouz, A. (2019) · Genome Biology · doi:10.1186/s13059-019-1795-z · Cited by 628
Crowell, H. L., Soneson, C., Germain, P.-L., Calini, D., Collin, L., Raposo, C., Malhotra, D., & Robinson, M. D. (2020) · Nature Communications · doi:10.1038/s41467-020-19894-4 · Cited by 515
Automated methods for cell type annotation on scRNA-seq data
Pasquini, G., Rojo Arias, J. E., Schäfer, P., & Busskamp, V. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.01.015 · Cited by 249
Metadata
- Authors
- Diaz-Mejia, J. J., Meng, E. C., Pico, A. R., MacParland, S. A., Ketela, T., Pugh, T. J., Bader, G. D., & Morris, J. H.
- Venue
- F1000Research
- Publisher
- F1000 Research Ltd
- Published
- 2019-03-15
- Volume
- 8
- Pages
- 296
- ISSN
- 2046-1402 (ISSN-L)
- OpenAlex
- W4256385088
- Cited by
- 52
- References
- 9
- FWCI
- 2.65
- Type
- journal article
- Language
- en
- Update notices
- Open access
- gold
- Funders
- Chan Zuckerberg Initiative (2018-183120)
- National Resource for Network Biology (P41GM103504)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.