Evaluation of single-cell classifiers for single-cell RNA sequencing data sets
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Single-cell RNA sequencing (scRNA-seq) has been rapidly developing and widely applied in biological and medical research. Identification of cell types in scRNA-seq data sets is an essential step before in-depth investigations of their functional and pathological roles. However, the conventional workflow based on clustering and marker genes is not scalable for an increasingly large number of scRNA-seq data sets due to complicated procedures and manual annotation. Therefore, a number of tools have been developed recently to predict cell types in new data sets using reference data sets. These methods have not been generally adapted due to a lack of tool benchmarking and user guidance. In this article, we performed a comprehensive and impartial evaluation of nine classification software tools specifically designed for scRNA-seq data sets. Results showed that Seurat based on random forest, SingleR based on correlation analysis and CaSTLe based on XGBoost performed better than others. A simple ensemble voting of all tools can improve the predictive accuracy. Under nonideal situations, such as small-sized and class-imbalanced reference data sets, tools based on cluster-level similarities have superior performance. However, even with the function of assigning ‘unassigned’ labels, it is still challenging to catch novel cell types by solely using any of the single-cell classifiers. This article provides a guideline for researchers to select and apply suitable classification tools in their analysis workflows and sheds some lights on potential direction of future improvement on classification tools.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Massively parallel digital transcriptional profiling of single cells
Zheng, G. X. Y., Terry, J. M., Belgrader, P., Ryvkin, P., Bent, Z. W., Wilson, R., Ziraldo, S. B., Wheeler, T. D., McDermott, G. P., Zhu, J., et al. (2017) · Nature Communications · doi:10.1038/ncomms14049 · Cited by 7,353
Spatial reconstruction of single-cell gene expression data
Satija, R., Farrell, J. A., Gennert, D., Schier, A. F., & Regev, A. (2015) · Nature Biotechnology · doi:10.1038/nbt.3192 · Cited by 7,055
Aran, D., Looney, A. P., Liu, L., Wu, E., Fong, V., Hsu, A., Chak, S., Naikawadi, R. P., Wolters, P. J., Abate, A. R., et al. (2019) · Nature Immunology · doi:10.1038/s41590-018-0276-y · Cited by 4,984 · cited 18×
“Based on the accuracy, ROC and other evaluation metrics, we demonstrated that Seurat, SingleR and CaSTLe outperformed the rest of tools.”
MethodInfluentialDroplet barcoding for single-cell transcriptomics applied to embryonic stem cells
Klein, A. M., Mazutis, L., Akartuna, I., Tallapragada, N., Veres, A., Li, V., Peshkin, L., Weitz, D. A., & Kirschner, M. W. (2015) · Cell · doi:10.1016/j.cell.2015.04.044 · Cited by 3,356
Single-cell transcriptomics of 20 mouse organs creates a Tabula Muris
The Tabula Muris Consortium, coordination, O., coordination, L., processing, O. C. A., sequencing, L. P. A., analysis, C. D., annotation, C. T., Writing group, Supplemental text writing group, & Principal investigators (2018) · Nature · doi:10.1038/s41586-018-0590-4 · Cited by 2,842
Batch effects in single-cell RNA-sequencing data are corrected by matching mutual nearest neighbors
Haghverdi, L., Lun, A. T. L., Morgan, M. D., & Marioni, J. C. (2018) · Nature Biotechnology · doi:10.1038/nbt.4091 · Cited by 2,416
Smart-seq2 for sensitive full-length transcriptome profiling in single cells
Picelli, S., Björklund, Å. K., Faridani, O. R., Sagasser, S., Winberg, G., & Sandberg, R. (2013) · Nature Methods · doi:10.1038/nmeth.2639 · Cited by 2,358
Regev, A., Teichmann, S. A., Lander, E. S., Amit, I., Benoist, C., Birney, E., Bodenmiller, B., Campbell, P., Carninci, P., Clatworthy, M., et al. (2017) · eLife · doi:10.7554/elife.27041 · Cited by 2,232
A comparison of single-cell trajectory inference methods
Saelens, W., Cannoodt, R., Todorov, H., & Saeys, Y. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0071-9 · Cited by 1,670
Mapping the Mouse Cell Atlas by Microwell-Seq
Han, X., Wang, R., Zhou, Y., Fei, L., Sun, H., Lai, S., Saadatpour, A., Zhou, Z., Chen, H., Ye, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.001 · Cited by 1,597
A Cancer Cell Program Promotes T Cell Exclusion and Resistance to Checkpoint Blockade
Jerby-Arnon, L., Shah, P., Cuoco, M. S., Rodman, C., Su, M.-J., Melms, J. C., Leeson, R., Kanodia, A., Mei, S., Lin, J.-R., et al. (2018) · Cell · doi:10.1016/j.cell.2018.09.006 · Cited by 1,429
Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric
Boughorbel, S., Jarray, F., & El-Anbari, M. (2017) · PLOS ONE · doi:10.1371/journal.pone.0177678 · Cited by 1,222 · cited 1×
“In addition, Matthews correlation coefficient (MCC) takes into account both true and false positives and negatives, which is regarded as a balanced measure for class-imbalanced data sets [41, 42] and defined as the following formula:”
MethodAltered human oligodendrocyte heterogeneity in multiple sclerosis
Jäkel, S., Agirre, E., Mendanha Falcão, A., van Bruggen, D., Lee, K. W., Knuesel, I., Malhotra, D., ffrench-Constant, C., Williams, A., & Castelo-Branco, G. (2019) · Nature · doi:10.1038/s41586-019-0903-2 · Cited by 854
Bias, robustness and scalability in single-cell differential expression analysis
Soneson, C., & Robinson, M. D. (2018) · Nature Methods · doi:10.1038/nmeth.4612 · Cited by 770
Single-cell analysis of mixed-lineage states leading to a binary cell fate choice
Olsson, A., Venkatasubramanian, M., Chaudhri, V. K., Aronow, B. J., Salomonis, N., Singh, H., & Grimes, H. L. (2016) · Nature · doi:10.1038/nature19348 · Cited by 476
Combinatorial labeling of single cells for gene expression cytometry
Fan, H. C., Fu, G. K., & Fodor, S. P. A. (2015) · Science · doi:10.1126/science.1258367 · Cited by 455
A Comparison of MCC and CEN Error Measures in Multi-Class Prediction
Jurman, G., Riccadonna, S., & Furlanello, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041882 · Cited by 406 · cited 1×
“In addition, Matthews correlation coefficient (MCC) takes into account both true and false positives and negatives, which is regarded as a balanced measure for class-imbalanced data sets [41, 42] and defined as the following formula:”
MethodMapping the Mouse Cell Atlas by Microwell-Seq
Han, X., Wang, R., Zhou, Y., Fei, L., Sun, H., Lai, S., Saadatpour, A., Zhou, Z., Chen, H., Ye, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.05.012 · Cited by 384
The emergent landscape of the mouse gut endoderm at single-cell resolution
Nowotschin, S., Setty, M., Kuo, Y.-Y., Liu, V., Garg, V., Sharma, R., Simon, C. S., Saiz, N., Gardner, R., Boutet, S. C., et al. (2019) · Nature · doi:10.1038/s41586-019-1127-1 · Cited by 383 · cited 1×
“In addition, similar tests were performed on an scRNA-seq data set of the mouse gut endoderm with >100,000 cells [46].”
MethodCrow, M., Paul, A., Ballouz, S., Huang, Z. J., & Gillis, J. (2018) · Nature Communications · doi:10.1038/s41467-018-03282-0 · Cited by 365
A systematic performance evaluation of clustering methods for single-cell RNA-seq data
Duò, A., Robinson, M. D., & Soneson, C. (2018) · F1000Research · doi:10.12688/f1000research.15666.2 · Cited by 296
Wang, T., Li, B., Nelson, C. E., & Nabavi, S. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-2599-6 · Cited by 286 · cited 1×
“Single-cell label prediction is a relatively new approach, unlike other analysis components in the scRNA-seq workflow such as differential expression [22, 23], clustering [24, 25], trajectory inference [26] and imputation [27], andhas not been evaluated systematically and comprehensively.”
MethodComparison of clustering tools in R for medium-sized 10x Genomics single-cell RNA-sequencing data
Freytag, S., Tian, L., Lönnstedt, I., Ng, M., & Bahlo, M. (2018) · F1000Research · doi:10.12688/f1000research.15809.1 · Cited by 220 · cited 2×
“Clustering-based approaches could generate near-truth cell type labels of mixed cell line data when the number of clusters is known [25].”
BackgroundMethod
Cited by
Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., & Yao, J. (2021) · doi:10.1101/2021.12.05.471261 · Cited by 617
Single-Cell Transcriptome Analysis in Plants: Advances and Challenges
Shaw, R., Tian, X., & Xu, J. (2021) · Molecular Plant · doi:10.1016/j.molp.2020.10.012 · Cited by 288
Automated methods for cell type annotation on scRNA-seq data
Pasquini, G., Rojo Arias, J. E., Schäfer, P., & Busskamp, V. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.01.015 · Cited by 249
“Another benchmark study, comparing different classes of tools, shows that combining multiple tools is highly encouraged for improving the accuracy [70].”
BackgroundComputational methods for the integrative analysis of single-cell data
Forcato, M., Romano, O., & Bicciato, S. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa042 · Cited by 71
Machine Learning for Predicting the 3-Year Risk of Incident Diabetes in Chinese Adults
Wu, Y., Hu, H., Cai, J., Chen, R., Zuo, X., Cheng, H., & Yan, D. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.626331 · Cited by 28
Metadata
- Authors
- Zhao, X., Wu, S., Fang, N., Sun, X., & Fan, J.
- Venue
- Briefings in Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-10-23
- Volume
- 21
- Issue
- 5
- Pages
- 1581-1595
- ISSN
- 1467-5463 (ISSN-L)
- 1477-4054
- OpenAlex
- W2984629040
- Cited by
- 97
- References
- 23
- FWCI
- 2.94
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.