CoChat
Sign Up
Cited by 136Single-cell and spatial transcriptomicsGene expression and cancer classificationGenomics and Phylogenetic Studies

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.

clustifyr: an R package for automated single-cell RNA sequencing cluster classification

Open access
Fu, Rui, Gillen, Austin E., Sheridan, Ryan M., Tian, Chengzhe, Daya, Michelle, Hao, Yue, Hesselberth, Jay R., Riemondy, Kent A.·2020·F1000Research
DOI 10.12688/f1000research.22969.2PMC PMC7383722
Open this paper in CoChat — free accountOpen PDF

Abstract

ABSTRACT Background In single-cell RNA sequencing (scRNA-seq) analysis, assignment of likely cell types remains a time-consuming, error-prone, and biased process. Current packages for identity assignment use limited types of reference data, and often have rigid data structure requirements. As such, a more flexible tool, capable of handling multiple types of reference data and data structures, would be beneficial. Findings To address difficulties in cluster identity assignment, we developed the clustifyr R package. The package leverages external datasets, including gene expression profiles from scRNA-seq, bulk RNA-seq, microarray expression data, and/or signature gene lists, to assign likely cell types. We benchmark various parameters of a correlation-based approach, and also implement a variety of gene list enrichment methods. By providing tools for exploratory data analysis, we demonstrate the feasibility of a simple and effective data-driven approach for cell type assignment in scRNA-seq cell clusters. Conclusions clustifyr is a lightweight and effective cell type assignment tool developed for compatibility with various scRNA-seq analysis workflows. clustifyr is publicly available at https://github.com/rnabioco/clustifyr

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.12688/f1000research.22969.2

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. Schematic for clustifyr input and output.
Figure 1. Schematic for clustifyr input and output.CC BY · PubMed Central
Figure 2. Parameter considerations for clustifyr. A) Comparison of median F1-scores of different correlation methods for classifying across platforms using the PBMC-bench dataset. B) Heatmap showing correlation coefficients between query cell types and the reference cell types from a rejection test, whereby megakaryocytes were excluded from the reference dataset. The Neg.Cell cluster is megakaryocytes, which is correctly not annotated a different cell type when megakaryocytes are not present in the reference. By default clusters with correlation < 0.50 are assigned as “unassigned” by clustifyr. C) Comparison of correlation coefficients with and without feature selection when comparing average gene expression per cell type between two pancreas scRNA-seq datasets. The “unclassified” cell type was not defined in the Segerstolpe et al dataset. D) Accuracy (defined as the ratio between the number of correctly classified clusters and the overall number of clusters) and performance were assessed with decreasing query cluster cell numbers using the Tabula Muris as the query dataset and the Mouse cell atlas as the reference dataset. E) Example of overclustering the query data and assigning cell types for data exploration. UMAP of PBMC dataset generated by 10x Genomics with cell types assigned by comparing to reference data from CBMC cells from Stoeckius et al. 2017. F) An assessment of the median F1-score when using single or multiple averaged profiles as reference cell types was conducted using the PBMC-bench test set. The number of reference expression profiles to generate for each cell type is determined by the number of cells in the cluster (n), and the sub-clustering power argument (x), with the formula n x.
Figure 2. Parameter considerations for clustifyr. A) Comparison of median F1-scores of different correlation methods for classifying across platforms using the PBMC-bench dataset. B) Heatmap showing correlation coefficients between query cell types and the reference cell types from a rejection test, whereby megakaryocytes were excluded from the reference dataset. The Neg.Cell cluster is megakaryocytes, which is correctly not annotated a different cell type when megakaryocytes are not present in the reference. By default clusters with correlation < 0.50 are assigned as “unassigned” by clustifyr. C) Comparison of correlation coefficients with and without feature selection when comparing average gene expression per cell type between two pancreas scRNA-seq datasets. The “unclassified” cell type was not defined in the Segerstolpe et al dataset. D) Accuracy (defined as the ratio between the number of correctly classified clusters and the overall number of clusters) and performance were assessed with decreasing query cluster cell numbers using the Tabula Muris as the query dataset and the Mouse cell atlas as the reference dataset. E) Example of overclustering the query data and assigning cell types for data exploration. UMAP of PBMC dataset generated by 10x Genomics with cell types assigned by comparing to reference data from CBMC cells from Stoeckius et al. 2017. F) An assessment of the median F1-score when using single or multiple averaged profiles as reference cell types was conducted using the PBMC-bench test set. The number of reference expression profiles to generate for each cell type is determined by the number of cells in the cluster (n), and the sub-clustering power argument (x), with the formula n x.CC BY · PubMed Central
Figure 3. clustifyr can utilize multiple reference data types. UMAP projections of PBMCs showing the ground truth cell types ( A), or cell types called by clustifyr using microarray data from sorted immune cell types ( B), bulk RNA-seq from immune cell populations ( C) or scRNA-seq data from CBMCs ( D).
Figure 3. clustifyr can utilize multiple reference data types. UMAP projections of PBMCs showing the ground truth cell types ( A), or cell types called by clustifyr using microarray data from sorted immune cell types ( B), bulk RNA-seq from immune cell populations ( C) or scRNA-seq data from CBMCs ( D).CC BY · PubMed Central
Figure 4. clustifyr accurately and rapidly annotates cell types. A) Accuracy and run-time of classifications generated by clustifyr or existing methods using the Tabula Muris dataset to benchmark cell type classifications between datasets generated with the Smart-Seq2 or 10x Genomics sequencing platforms. Each point represents a different tissue comparison. clustifyr (m3drop) indicates clustifyr run using variable genes defined by M3drop, clustifyr_lists (hyper) uses hypergeometric tests to compare marker gene lists, and clustifyr_lists(jaccard) calculates the jaccard index between marker gene lists to annotate cell types. B) Performance comparison of clustifyr to existing methods with random subsamples of cells from the Smart-Seq2 Tabula Muris dataset. Error bars represent standard error of the mean and are derived from 5 independent subsamples of the dataset. C) Performance comparison of clustifyr to existing methods testing classification of an Allen Institute Brain Atlas dataset from two murine brain regions that contain 34 cell types. scPred is not shown as it failed with an error on this dataset. D) Comparing clustifyr to existing methods for rejecting unseen populations using PBMC data. Three reference PBMC datasets were generated that excluded either T-cells, CD4+ T-cells or memory T-cells respectively. The % of rejected indicates the % of the indicated cell type that was not misclassified when the cell type was missing from the reference.
Figure 4. clustifyr accurately and rapidly annotates cell types. A) Accuracy and run-time of classifications generated by clustifyr or existing methods using the Tabula Muris dataset to benchmark cell type classifications between datasets generated with the Smart-Seq2 or 10x Genomics sequencing platforms. Each point represents a different tissue comparison. clustifyr (m3drop) indicates clustifyr run using variable genes defined by M3drop, clustifyr_lists (hyper) uses hypergeometric tests to compare marker gene lists, and clustifyr_lists(jaccard) calculates the jaccard index between marker gene lists to annotate cell types. B) Performance comparison of clustifyr to existing methods with random subsamples of cells from the Smart-Seq2 Tabula Muris dataset. Error bars represent standard error of the mean and are derived from 5 independent subsamples of the dataset. C) Performance comparison of clustifyr to existing methods testing classification of an Allen Institute Brain Atlas dataset from two murine brain regions that contain 34 cell types. scPred is not shown as it failed with an error on this dataset. D) Comparing clustifyr to existing methods for rejecting unseen populations using PBMC data. Three reference PBMC datasets were generated that excluded either T-cells, CD4+ T-cells or memory T-cells respectively. The % of rejected indicates the % of the indicated cell type that was not misclassified when the cell type was missing from the reference.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 33 references

Research works printed but not linked: 1

  1. R Fu, 2020, rnabioco/clustifyr 0.99.7 (Version 0.99.7)., Zenodo.

Cited by

Sorted by citations

See all 136

Related

Metadata

Authors
Fu, R., Gillen, A. E., Sheridan, R. M., Tian, C., Daya, M., Hao, Y., Hesselberth, J. R., & Riemondy, K. A.
Affiliations
  • University of Colorado Anschutz (US)
  • University of Colorado Boulder (US)
  • North Carolina State University (US)
Corresponding author
Riemondy, Kent A.
ORCID
Venue
F1000Research
Publisher
F1000 Research Ltd
Published
2020-07-16
Volume
9
Pages
223
ISSN
2046-1402 (ISSN-L)
DOI
10.12688/f1000research.22969.2
PMC
PMC7383722
OpenAlex
W3014185549
Cited by
135
References
33
Citations by year
2026: 13 · 2025: 34 · 2024: 27 · 2023: 25 · 2022: 19 · 2021: 14 · 2020: 3
FWCI
4.95
Type
journal article
Language
en
Open access
Licence
Locations
  • F1000Research (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institute of General Medical Sciences (R35GM119550)
  • University of Colorado School of Medicine (RNA Bioscience Initiative)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
Keywords
single-cell RNA sequencing, cell type classification, reference data, gene expression profiles, bulk RNA-seq, gene expression microarrays, R package
MeSH terms
Humans, Software, Sequence Analysis, RNA, RNA, Small Cytoplasmic, Gene Expression Profiling, Single-Cell Analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review