Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index; 12 of the 34 sources its claims cite were checked for support.

scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data

Open access

Preprint — published as 10.1038/s42256-022-00534-z · Paper page

Abstract

Annotating cell types based on the single-cell RNA-seq data is a prerequisite for researches on disease progress and tumor microenvironment. Here we show existing annotation methods typically suffer from lack of curated marker gene lists, improper handling of batch effect, and difficulty in leveraging the latent gene-gene interaction information, impairing their generalization and robustness. We developed a pre-trained deep neural network-based model scBERT (single-cell Bidirectional Encoder Representations from Transformers) to overcome the challenges. Following BERT’s approach of pre-train and fine-tune, scBERT obtains a general understanding of gene-gene interaction by being pre-trained on huge amounts of unlabeled scRNA-seq data and is transferred to the cell type annotation task of unseen and user-specific scRNA-seq data for supervised fine-tuning. Extensive and rigorous benchmark studies validated the superior performance of scBERT on cell type annotation, novel cell type discovery, robustness to batch effect, and model interpretability.

Overview

CoChat Summary

TL;DR

scBERT is a pretrained deep language model adapted from BERT/Performer for scRNA-seq cell type annotation, using self-supervised pretraining on unlabeled data and supervised fine-tuning. It reports superior accuracy, robustness to batch effects, novel cell type discovery, and gene-level interpretability across multiple benchmarks.

Key findings

  • Across 9 scRNA-seq datasets covering 17 major organs/tissues, more than 50 cell types, and over 500,000 cells, scBERT surpassed comparison methods in accuracy and macro F1 score on most datasets.
  • On the Zheng68K PBMC dataset, scBERT achieved F1=0.691 and accuracy=0.759 versus best other method F1=0.659 and accuracy=0.704; for highly similar CD8+ cytotoxic T and CD8+/CD45RA+ T cells, F1=0.788 vs. 0.617 (P=9.025e-5) and accuracy=0.801 vs. 0.724 (P=2.265e-5).
  • In cross-cohort pancreas benchmarks, scBERT reached accuracy=0.992 versus scNym=0.904, SciBet=0.985, Seurat=0.984, and SingleR=0.987; it correctly annotated >97% of cells in Muraro and >99% in the other three datasets.
  • For novel cell type discovery, scBERT achieved novel-class accuracy=0.329 vs. SciBet=0.174 and scmap_cluster=0.174, and known-class accuracy=0.942 vs. SciBet=0.784 and scmap_cluster=0.666; scBERT embedding clustering gave ARI=0.95 vs. 0.87 for raw expression.

Methods

  • Model: BERT-style pretrain/fine-tune paradigm with a Performer encoder (6 layers, 10 heads), gene2vec gene embeddings, expression binning into 200-dimensional vectors, self-supervised masking of non-zero expression for reconstruction, and supervised fine-tuning with a 1D convolution plus 3-layer neural network classifier.
  • Data: pretraining on PanglaoDB (209 human single-cell datasets, 74 tissues, 1,126,580 cells) and a large heart dataset (451,513 cells) for hyperparameter sensitivity; benchmarking used 9 datasets, 5-fold cross-validation, leave-one-dataset-out pancreas tests, class-imbalanced PBMC tests, MacParland novel cell type tests, and cross-organ Human Cell Atlas tests.
  • Analysis: accuracy and macro F1 score, Wilcoxon tests, adjusted Rand index, and enrichment analysis; comparisons included marker-based, correlation-based, and ML-based methods.

Limitations

  • Stated: expression binning sacrifices some data resolution; gene interaction network prior knowledge is not explicitly incorporated; masking only non-zero values may lower utilization of single-cell data for pretraining; and there is no unified quantitative evaluation metric for novel cell type detection.
  • Evident from reported results: novel cell type accuracy remains low (0.329), and cross-organ performance was only on par with comparison methods, indicating limited gains in those settings.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1101/2021.12.05.471261

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

6 Valid · 2 Not valid · 18 Not checked · 84 Own results and interpretations

53 more claims · 3 with their cited sources checked

Create a free account to see all 110 claims and to verify the 22 references still open.

See all claims — free account
  • It may state two separate claims as one. 35
  • Probably not a substantial claim. 19

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 68 references

Research works printed but not linked: 7

  1. 10.1038/s41467-020-16839-9 · DOI · Paper page
  2. 10.1038/s41467-020-16839-9 · DOI · Paper page
  3. 10.1186/s13059-019-1727-y · DOI · Paper page
  4. 10.1038/s41467-020-16839-9 · DOI · Paper page
  5. 10.1186/s13059-019-1727-y · DOI · Paper page
  6. Do Transformers Really Perform Badly for Graph Representation? https://proceedings.neurips.cc/paper/2021/hash/f1c1592588411002af340cbaedd6fc33-Abstract.html.
  7. 10.1186/s13059-020-1935-5 · DOI · Paper page

Cited by

See all 704

Metadata

Authors
Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., & Yao, J.
Affiliations
  • Tencent (China) (CN)
  • Shanghai Jiao Tong University (CN)
  • Harvard University (US)
  • The University of Texas at Arlington (US)
Corresponding author
  • Lu, Hui
  • Yao, Jianhua
Venue
Nature Machine Intelligence
Publisher
openRxiv
Published
2021-12-07
Volume
4
Issue
10
Pages
852-866
ISSN
2522-5839 (ISSN-L)
OpenAlex
W4297243391
Cited by
617
References
68
Citations by year
2026: 172 · 2025: 233 · 2024: 158 · 2023: 52 · 2022: 2
FWCI
36.41
Type
journal article
Language
en
Open access
Locations
  • Nature Machine Intelligence (publishedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
Funders
  • Yale University
  • Shanghai Jiao Tong University
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
Keywords
cell type annotation, single-cell RNA sequencing, transformer models, self-supervised pre-training, gene-gene interaction, marker genes, fine-tuning, model interpretability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data. https://app.cochat.ai/papers/10.1101/2021.12.05.471261
The paper
Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., & Yao, J. (2021). scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data. Nature Machine Intelligence. https://doi.org/10.1101/2021.12.05.471261

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.