Deep learning does not outperform classical machine learning for cell-type annotation
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Deep learning has revolutionized image analysis and natural language processing with remarkable accuracies in prediction tasks, such as image labeling and semantic segmentation or named-entity recognition and semantic role labeling. Specifically, the combination of algorithmic and hardware advances with the appearance of large and well-labeled datasets has led up to seminal contributions in these fields. The emergence of large amounts of data from single-cell RNA-seq and the recent global effort to chart all cell types in the Human Cell Atlas has attracted an interest in deep-learning applications. However, all current approaches are unsupervised, i.e. , learning of latent spaces without using any cell labels, even though supervised learning approaches are often more powerful in feature learning and the most popular approach in the current AI revolution by far. Here, we ask why this is the case. In particular we ask whether supervised deep learning can be used for cell annotation, i.e. to predict cell-type labels from single-cell gene expression profiles. After evaluating 10 classification methods across 14 datasets, we notably find that deep learning does not outperform classical machine-learning methods in the task. Thus, cell-type prediction based on gene-signature derived cell-type labels is potentially too simplistic a task for complex non-linear methods, which demands better labels of functional single-cell readouts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Aran, D., Looney, A. P., Liu, L., Wu, E., Fong, V., Hsu, A., Chak, S., Naikawadi, R. P., Wolters, P. J., Abate, A. R., et al. (2019) · Nature Immunology · doi:10.1038/s41590-018-0276-y · Cited by 4,984
Simultaneous epitope and transcriptome measurement in single cells
Stoeckius, M., Hafemeister, C., Stephenson, W., Houck-Loomis, B., Chattopadhyay, P. K., Swerdlow, H., Satija, R., & Smibert, P. (2017) · Nature Methods · doi:10.1038/nmeth.4380 · Cited by 3,098
Single-cell transcriptomics of 20 mouse organs creates a Tabula Muris
The Tabula Muris Consortium, coordination, O., coordination, L., processing, O. C. A., sequencing, L. P. A., analysis, C. D., annotation, C. T., Writing group, Supplemental text writing group, & Principal investigators (2018) · Nature · doi:10.1038/s41586-018-0590-4 · Cited by 2,842
Molecular Architecture of the Mouse Nervous System
Zeisel, A., Hochgerner, H., Lönnerberg, P., Johnsson, A., Memic, F., van der Zwan, J., Häring, M., Braun, E., Borm, L. E., La Manno, G., et al. (2018) · Cell · doi:10.1016/j.cell.2018.06.021 · Cited by 2,552
Deep generative modeling for single-cell transcriptomics
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., & Yosef, N. (2018) · Nature Methods · doi:10.1038/s41592-018-0229-2 · Cited by 2,530 · cited 1×
“While waiting for the emergence of such labels, the scientific community has developed a series of unsupervised deep-learning approaches, including batch effect correction 37 , large scale data integration 38 , single-cell perturbation prediction 39 , and dropout effects denoising 40 .”
BackgroundBatch effects in single-cell RNA-sequencing data are corrected by matching mutual nearest neighbors
Haghverdi, L., Lun, A. T. L., Morgan, M. D., & Marioni, J. C. (2018) · Nature Biotechnology · doi:10.1038/nbt.4091 · Cited by 2,416
Dixit, A., Parnas, O., Li, B., Chen, J., Fulco, C. P., Jerby-Arnon, L., Marjanovic, N. D., Dionne, D., Burks, T., Raychowdhury, R., et al. (2016) · Cell · doi:10.1016/j.cell.2016.11.038 · Cited by 1,863
Single-Cell Transcriptome Profiling of Human Pancreatic Islets in Health and Type 2 Diabetes
Segerstolpe, Å., Palasantza, A., Eliasson, P., Andersson, E.-M., Andréasson, A.-C., Sun, X., Picelli, S., Sabirsh, A., Clausen, M., Bjursell, M. K., et al. (2016) · Cell Metabolism · doi:10.1016/j.cmet.2016.08.020 · Cited by 1,550 · cited 2×
“When we used tissue studies of the human pancreas 11–14 and the mouse brain 15–18 , all machine-learning models and cellFishing.jl performed equally well on the prediction task.”
Method
Cited by
Cross-tissue immune cell analysis reveals tissue-specific features in humans
Domínguez Conde, C., Xu, C., Jarvis, L. B., Rainbow, D. B., Wells, S. B., Gomes, T., Howlett, S. K., Suchanek, O., Polanski, K., King, H. W., et al. (2022) · Science · doi:10.1126/science.abl5197 · Cited by 1,159
A comparison of automatic cell identification methods for single-cell RNA sequencing data
Abdelaal, T., Michielsen, L., Cats, D., Hoogduin, D., Mei, H., Reinders, M. J. T., & Mahfouz, A. (2019) · Genome Biology · doi:10.1186/s13059-019-1795-z · Cited by 628
The single-cell eQTLGen consortium
van der Wijst, M., de Vries, D., Groot, H., Trynka, G., Hon, C., Bonder, M., Stegle, O., Nawijn, M., Idaghdour, Y., van der Harst, P., et al. (2020) · eLife · doi:10.7554/elife.52155 · Cited by 218
Self-citationFortelny, N., & Bock, C. (2020) · Genome Biology · doi:10.1186/s13059-020-02100-5 · Cited by 135
clustifyr: an R package for automated single-cell RNA sequencing cluster classification
Fu, R., Gillen, A. E., Sheridan, R. M., Tian, C., Daya, M., Hao, Y., Hesselberth, J. R., & Riemondy, K. A. (2020) · F1000Research · doi:10.12688/f1000research.22969.2 · Cited by 135
Enhancing scientific discoveries in molecular biology with deep generative models
Lopez, R., Gayoso, A., & Yosef, N. (2020) · Molecular Systems Biology · doi:10.15252/msb.20199198 · Cited by 82
“In a disentangled representation, each dimension in a represents variation due to one notable underlying source of variation (e.g., cell cycle). predicting broad cell types in scRNA-seq (preprint: Ko¨hler et al , 2019).”
BackgroundSfaira accelerates data and model reuse in single cell genomics
Fischer, D. S., Dony, L., König, M., Moeed, A., Zappia, L., Heumos, L., Tritschler, S., Holmberg, O., Aliee, H., & Theis, F. J. (2021) · Genome Biology · doi:10.1186/s13059-021-02452-6 · Cited by 33
Self-citation
Metadata
- Authors
- Köhler, N. D., Büttner, M., Andriamanga, N., & Theis, F. J.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2019-05-31
- OpenAlex
- W2947809439
- Cited by
- 31
- References
- 26
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.