Deep learning does not outperform classical machine learning for cell-type annotation
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Deep learning has revolutionized image analysis and natural language processing with remarkable accuracies in prediction tasks, such as image labeling and semantic segmentation or named-entity recognition and semantic role labeling. Specifically, the combination of algorithmic and hardware advances with the appearance of large and well-labeled datasets has led up to seminal contributions in these fields. The emergence of large amounts of data from single-cell RNA-seq and the recent global effort to chart all cell types in the Human Cell Atlas has attracted an interest in deep-learning applications. However, all current approaches are unsupervised, i.e. , learning of latent spaces without using any cell labels, even though supervised learning approaches are often more powerful in feature learning and the most popular approach in the current AI revolution by far. Here, we ask why this is the case. In particular we ask whether supervised deep learning can be used for cell annotation, i.e. to predict cell-type labels from single-cell gene expression profiles. After evaluating 10 classification methods across 14 datasets, we notably find that deep learning does not outperform classical machine-learning methods in the task. Thus, cell-type prediction based on gene-signature derived cell-type labels is potentially too simplistic a task for complex non-linear methods, which demands better labels of functional single-cell readouts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Aran, D., Looney, A. P., Liu, L., Wu, E., Fong, V., Hsu, A., Chak, S., Naikawadi, R. P., Wolters, P. J., Abate, A. R., et al. (2019) · Nature Immunology · doi:10.1038/s41590-018-0276-y · Cited by 4,984
Simultaneous epitope and transcriptome measurement in single cells
Stoeckius, M., Hafemeister, C., Stephenson, W., Houck-Loomis, B., Chattopadhyay, P. K., Swerdlow, H., Satija, R., & Smibert, P. (2017) · Nature Methods · doi:10.1038/nmeth.4380 · Cited by 3,098
Single-cell transcriptomics of 20 mouse organs creates a Tabula Muris
The Tabula Muris Consortium, coordination, O., coordination, L., processing, O. C. A., sequencing, L. P. A., analysis, C. D., annotation, C. T., Writing group, Supplemental text writing group, & Principal investigators (2018) · Nature · doi:10.1038/s41586-018-0590-4 · Cited by 2,842
Molecular Architecture of the Mouse Nervous System
Zeisel, A., Hochgerner, H., Lönnerberg, P., Johnsson, A., Memic, F., van der Zwan, J., Häring, M., Braun, E., Borm, L. E., La Manno, G., et al. (2018) · Cell · doi:10.1016/j.cell.2018.06.021 · Cited by 2,552
Deep generative modeling for single-cell transcriptomics
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., & Yosef, N. (2018) · Nature Methods · doi:10.1038/s41592-018-0229-2 · Cited by 2,530 · cited 1×
“While waiting for the emergence of such labels, the scientific community has developed a series of unsupervised deep-learning approaches, including batch effect correction 37 , large scale data integration 38 , single-cell perturbation prediction 39 , and dropout effects denoising 40 .”
BackgroundBatch effects in single-cell RNA-sequencing data are corrected by matching mutual nearest neighbors
Haghverdi, L., Lun, A. T. L., Morgan, M. D., & Marioni, J. C. (2018) · Nature Biotechnology · doi:10.1038/nbt.4091 · Cited by 2,416
Dixit, A., Parnas, O., Li, B., Chen, J., Fulco, C. P., Jerby-Arnon, L., Marjanovic, N. D., Dionne, D., Burks, T., Raychowdhury, R., et al. (2016) · Cell · doi:10.1016/j.cell.2016.11.038 · Cited by 1,863
Single-Cell Transcriptome Profiling of Human Pancreatic Islets in Health and Type 2 Diabetes
Segerstolpe, Å., Palasantza, A., Eliasson, P., Andersson, E.-M., Andréasson, A.-C., Sun, X., Picelli, S., Sabirsh, A., Clausen, M., Bjursell, M. K., et al. (2016) · Cell Metabolism · doi:10.1016/j.cmet.2016.08.020 · Cited by 1,550 · cited 2×
“When we used tissue studies of the human pancreas 11–14 and the mouse brain 15–18 , all machine-learning models and cellFishing.jl performed equally well on the prediction task.”
MethodMolecular diversity and specializations among the cells of the adult mouse brain
Saunders, A., Macosko, E. Z., Wysoker, A., Goldman, M., Krienen, F. M., de Rivera, H., Bien, E., Baum, M., Bortolin, L., Wang, S., et al. (2018) · Cell · doi:10.1016/j.cell.2018.07.028 · Cited by 1,531 · cited 2×
“20,000 cells), and mouse brain 15–18 (approx.”
MethodBaron, M., Veres, A., Wolock, S. L., Faust, A. L., Gaujoux, R., Vetere, A., Ryu, J. H., Wagner, B. K., Shen-Orr, S. S., Klein, A. M., et al. (2016) · Cell Systems · doi:10.1016/j.cels.2016.08.011 · Cited by 1,508
Benchmarking atlas-level data integration in single-cell genomics
Luecken, M. D., Büttner, M., Chaichoompu, K., Danese, A., Interlandi, M., Mueller, M. F., Strobl, D. C., Zappia, L., Dugas, M., Colomé-Tatché, M., et al. (2022) · Nature Methods · doi:10.1038/s41592-021-01336-8 · Cited by 1,438
Self-citationSingle-cell profiling of the developing mouse brain and spinal cord with split-pool barcoding
Rosenberg, A. B., Roco, C. M., Muscat, R. A., Kuchina, A., Sample, P., Yao, Z., Graybuck, L. T., Peeler, D. J., Mukherjee, S., Chen, W., et al. (2018) · Science · doi:10.1126/science.aam8999 · Cited by 1,422 · cited 2×
“20,000 cells), and mouse brain 15–18 (approx.”
MethodSingle-Cell RNA-Seq Reveals Dynamic, Random Monoallelic Gene Expression in Mammalian Cells
Deng, Q., Ramsköld, D., Reinius, B., & Sandberg, R. (2014) · Science · doi:10.1126/science.1245316 · Cited by 1,345
Pooled CRISPR screening with single-cell transcriptome readout.
Datlinger, P., Rendeiro, A. F., Schmidl, C., Krausgruber, T., Traxler, P., Klughammer, J., Schuster, L. C., Kuchler, A., Alpar, D., & Bock, C. (2017) · Nature Methods · doi:10.1038/nmeth.4177 · Cited by 1,096 · cited 1×
“Such complex functional relationships are encountered between the cellular state (cell type or cluster identity) and functional assays, responses to drug treatment or gene knockouts 29,30 .”
BackgroundSingle-cell RNA-seq denoising using a deep count autoencoder
Eraslan, G., Simon, L. M., Mircea, M., Mueller, N. S., & Theis, F. J. (2019) · Nature Communications · doi:10.1038/s41467-018-07931-2 · Cited by 1,084 · cited 1×
“While waiting for the emergence of such labels, the scientific community has developed a series of unsupervised deep-learning approaches, including batch effect correction 37 , large scale data integration 38 , single-cell perturbation prediction 39 , and dropout effects denoising 40 .”
BackgroundSelf-citationGenetic programs in human and mouse early embryos revealed by single-cell RNA sequencing
Xue, Z., Huang, K., Cai, C., Cai, L., Jiang, C.-Y., Feng, Y., Liu, Z., Zeng, Q., Cheng, L., Sun, Y. E., et al. (2013) · Nature · doi:10.1038/nature12364 · Cited by 963
Single-cell mapping of gene expression landscapes and lineage in the zebrafish embryo
Wagner, D. E., Weinreb, C., Collins, Z. M., Briggs, J. A., Megason, S. G., & Klein, A. M. (2018) · Science · doi:10.1126/science.aar4362 · Cited by 849
scGen predicts single-cell perturbation responses
Lotfollahi, M., Wolf, F. A., & Theis, F. J. (2019) · Nature Methods · doi:10.1038/s41592-019-0494-8 · Cited by 661
Self-citationA comparison of automatic cell identification methods for single-cell RNA sequencing data
Abdelaal, T., Michielsen, L., Cats, D., Hoogduin, D., Mei, H., Reinders, M. J. T., & Mahfouz, A. (2019) · Genome Biology · doi:10.1186/s13059-019-1795-z · Cited by 628 · cited 1×
“Notably, recent studies for cell type annotation tasks did not split by dataset for evaluating the prediction accuracy, therefore limiting the informative power 19–21 .”
BackgroundAken, B. L., Achuthan, P., Akanni, W., Amode, M. R., Bernsdorff, F., Bhai, J., Billis, K., Carvalho-Silva, D., Cummins, C., Clapham, P., et al. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw1104 · Cited by 512
Supervised classification enables rapid annotation of cell atlases
Pliner, H. A., Shendure, J., & Trapnell, C. (2019) · doi:10.1101/538652 · Cited by 498
Heterogeneity in Oct4 and Sox2 Targets Biases Cell Fate in 4-Cell Mouse Embryos
Goolam, M., Scialdone, A., Graham, S. J. L., Macaulay, I. C., Jedrusik, A., Hupalowska, A., Voet, T., Marioni, J. C., & Zernicka-Goetz, M. (2016) · Cell · doi:10.1016/j.cell.2016.01.047 · Cited by 484
Molecular recording of mammalian embryogenesis
Chan, M. M., Smith, Z. D., Grosswendt, S., Kretzmer, H., Norman, T. M., Adamson, B., Jost, M., Quinn, J. J., Yang, D., Jones, M. G., et al. (2019) · Nature · doi:10.1038/s41586-019-1184-5 · Cited by 364 · cited 1×
“Furthermore, complex labels such as cell fates require creative experimental methods for tracing over several generations in both cell lines and complete animals, such as traceable genetic manipulation 31,32 .”
scMatch: a single-cell gene expression profile annotation tool using reference datasets
Hou, R., Denisenko, E., & Forrest, A. R. R. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz292 · Cited by 130
Cited by
Cross-tissue immune cell analysis reveals tissue-specific features in humans
Domínguez Conde, C., Xu, C., Jarvis, L. B., Rainbow, D. B., Wells, S. B., Gomes, T., Howlett, S. K., Suchanek, O., Polanski, K., King, H. W., et al. (2022) · Science · doi:10.1126/science.abl5197 · Cited by 1,159
A comparison of automatic cell identification methods for single-cell RNA sequencing data
Abdelaal, T., Michielsen, L., Cats, D., Hoogduin, D., Mei, H., Reinders, M. J. T., & Mahfouz, A. (2019) · Genome Biology · doi:10.1186/s13059-019-1795-z · Cited by 628
The single-cell eQTLGen consortium
van der Wijst, M., de Vries, D., Groot, H., Trynka, G., Hon, C., Bonder, M., Stegle, O., Nawijn, M., Idaghdour, Y., van der Harst, P., et al. (2020) · eLife · doi:10.7554/elife.52155 · Cited by 218
Self-citationFortelny, N., & Bock, C. (2020) · Genome Biology · doi:10.1186/s13059-020-02100-5 · Cited by 135
clustifyr: an R package for automated single-cell RNA sequencing cluster classification
Fu, R., Gillen, A. E., Sheridan, R. M., Tian, C., Daya, M., Hao, Y., Hesselberth, J. R., & Riemondy, K. A. (2020) · F1000Research · doi:10.12688/f1000research.22969.2 · Cited by 135
Metadata
- Authors
- Köhler, N. D., Büttner, M., Andriamanga, N., & Theis, F. J.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2019-05-31
- OpenAlex
- W2947809439
- Cited by
- 31
- References
- 26
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.