The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index; 12 of the 34 sources its claims cite were checked for support.
scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data
Open accessPreprint — published as 10.1038/s42256-022-00534-z · Paper page
Abstract
Annotating cell types based on the single-cell RNA-seq data is a prerequisite for researches on disease progress and tumor microenvironment. Here we show existing annotation methods typically suffer from lack of curated marker gene lists, improper handling of batch effect, and difficulty in leveraging the latent gene-gene interaction information, impairing their generalization and robustness. We developed a pre-trained deep neural network-based model scBERT (single-cell Bidirectional Encoder Representations from Transformers) to overcome the challenges. Following BERT’s approach of pre-train and fine-tune, scBERT obtains a general understanding of gene-gene interaction by being pre-trained on huge amounts of unlabeled scRNA-seq data and is transferred to the cell type annotation task of unseen and user-specific scRNA-seq data for supervised fine-tuning. Extensive and rigorous benchmark studies validated the superior performance of scBERT on cell type annotation, novel cell type discovery, robustness to batch effect, and model interpretability.
Overview
CoChat Summary
TL;DR
scBERT is a pretrained deep language model adapted from BERT/Performer for scRNA-seq cell type annotation, using self-supervised pretraining on unlabeled data and supervised fine-tuning. It reports superior accuracy, robustness to batch effects, novel cell type discovery, and gene-level interpretability across multiple benchmarks.
Key findings
- Across 9 scRNA-seq datasets covering 17 major organs/tissues, more than 50 cell types, and over 500,000 cells, scBERT surpassed comparison methods in accuracy and macro F1 score on most datasets.
- On the Zheng68K PBMC dataset, scBERT achieved F1=0.691 and accuracy=0.759 versus best other method F1=0.659 and accuracy=0.704; for highly similar CD8+ cytotoxic T and CD8+/CD45RA+ T cells, F1=0.788 vs. 0.617 (P=9.025e-5) and accuracy=0.801 vs. 0.724 (P=2.265e-5).
- In cross-cohort pancreas benchmarks, scBERT reached accuracy=0.992 versus scNym=0.904, SciBet=0.985, Seurat=0.984, and SingleR=0.987; it correctly annotated >97% of cells in Muraro and >99% in the other three datasets.
- For novel cell type discovery, scBERT achieved novel-class accuracy=0.329 vs. SciBet=0.174 and scmap_cluster=0.174, and known-class accuracy=0.942 vs. SciBet=0.784 and scmap_cluster=0.666; scBERT embedding clustering gave ARI=0.95 vs. 0.87 for raw expression.
Methods
- Model: BERT-style pretrain/fine-tune paradigm with a Performer encoder (6 layers, 10 heads), gene2vec gene embeddings, expression binning into 200-dimensional vectors, self-supervised masking of non-zero expression for reconstruction, and supervised fine-tuning with a 1D convolution plus 3-layer neural network classifier.
- Data: pretraining on PanglaoDB (209 human single-cell datasets, 74 tissues, 1,126,580 cells) and a large heart dataset (451,513 cells) for hyperparameter sensitivity; benchmarking used 9 datasets, 5-fold cross-validation, leave-one-dataset-out pancreas tests, class-imbalanced PBMC tests, MacParland novel cell type tests, and cross-organ Human Cell Atlas tests.
- Analysis: accuracy and macro F1 score, Wilcoxon tests, adjusted Rand index, and enrichment analysis; comparisons included marker-based, correlation-based, and ML-based methods.
Limitations
- Stated: expression binning sacrifices some data resolution; gene interaction network prior knowledge is not explicitly incorporated; masking only non-zero values may lower utilization of single-cell data for pretraining; and there is no unified quantitative evaluation metric for novel cell type detection.
- Evident from reported results: novel cell type accuracy remains low (0.329), and cross-organ performance was only on par with comparison methods, indicating limited gains in those settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
6 Valid · 2 Not valid · 18 Not checked · 84 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These methods are potentially affected by the batch effect across platforms and experiments 12 .”
§ scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data > Abstract
What the sources say
- Partly supportedcites [12] sourceExperimental — accuracy not yet measured
Our copy of Haghverdi et al. (2018) supports the general premise: the Introduction states that data generated 'in multiple laboratories using different cell dissociation and handling protocols, library preparation technologies and/or sequencing platforms' produce batch effects that 'can mask underlying biology or introduce spurious structure... and must be corrected prior to further analysis to avoid misleading conclusions', and the Discussion warns that uncorrected batch effects lead to 'batch-specific clusters... incorrectly interpreted as distinct cell types'. However, no passage addresses correlation-based annotation methods as such — the paper's critiques of existing approaches target linear-regression correction (limma removeBatchEffect, ComBat, RUVseq, svaseq) and reference-landmark projection strategies [11][12], not similarity/correlation-based cell-type annotation. The 'across platforms and experiments' qualifier is therefore sourced, but the attribution of this vulnerability specifically to correlation-based annotation is not stated in this paper. (LLM, conf=0.68)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Besides, most cell types are determined by a set of genes instead of a single marker gene 8 .”
§ scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data > Abstract
What the sources say
- Supportedcites [8] Moffitt, J. R. et al. Molecular, spatial, and functional single-cell profiling of the hypothalamic preoptic region. Science (1979) 362, (2018). source
The Results section ('scRNA-seq of the preoptic region') states directly: 'Notably, most clusters were discriminated by combinations of genes, rather than by single markers.' The Discussion repeats the point at the level of cell types: 'Many of these neuronal populations were defined by a combination of multiple genes, indicating that genetic intersectional approaches will be most useful in the functional interrogation of specific cell types.' The source's data are from ~70 neuronal clusters of the mouse preoptic region (plus major non-neuronal cell classes profiled by scRNA-seq/MERFISH), so the general phrasing 'most cell types' is a mild widening of scope, but the substance of the claim — cell types distinguished by gene sets rather than one marker gene — is what our copy asserts. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, HVGs are variable across different batches and different datasets, hindering their generalization ability across cohorts 16 .”
§ scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data > Abstract
What the sources say
- Could not verifycites [16] sourceExperimental — accuracy not yet measured
Our copy of Wick et al. (2019) is entirely about Oxford Nanopore basecalling performance (Albacore/Guppy/Chiron/Scrappie) on bacterial genomes; no passage mentions highly variable genes, single-cell data, or batch/dataset-driven feature selection. Searches for 'highly variable genes batch effect generalization across cohorts' and for variable-feature selection across datasets returned only the paper's own generalisation results (Results > Custom model performance, p. n/a: custom-Kp benefits confined to Enterobacteriaceae, mean consensus qscore change -1.64 outside it; Conclusions: basecalling models trained on the same species or close relative perform best) — an analogous but different point about basecaller training data and Dcm methylation, not about HVGs. Because our copy does not address the claim's subject, it can neither support nor contradict it; the closest analogue is the taxon-specific generalisation limitation in 'Custom model performance', which concerns basecalling models rather than variable-gene selection. (LLM, conf=0.72)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, even some well-documented marker genes do not have high expression in all the cells in the corresponding cell types 11 .”
§ scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data > Abstract
What the sources say
- Supportedcites [11] source
Section 2.2.1 ('Marker genes are unreliable due to sparsity') examines curated, well-established marker genes for CD4, CD14, CD8 and NK cells and states that 'we observe 0 cells with non-zero counts even for marker genes that are known to be expressed', with non-zero counts in as few as 4% of cells (and a maximum of 38% among CD4 markers), concluding that 'most marker genes are not definitively and specifically present in scRNA-seq data.' Figure 2's caption likewise shows canonical marker genes 'are unlikely to be observed' across cells of their cell type. This matches the paraphrased claim that well-documented markers are not highly expressed in all cells of the corresponding type. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Selecting HVGs ignores co-occurrence and biological interaction of genes (especially HVG and non-HVG), which are useful for cell type annotation 20 .”
§ scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data > Abstract
What the sources say
- Could not verifycites [20] sourceExperimental — accuracy not yet measured
The Discussion addresses relationships among plant leaf traits, and Methods > Data modifications describes species-level observations; neither addresses gene selection or cell-type annotation. Searches for HVGs, gene co-occurrence, biological interaction, and cell-type annotation found no relevant passage in our copy. Page numbers were not provided. (LLM, conf=0.78)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
53 more claims · 3 with their cited sources checked
Create a free account to see all 110 claims and to verify the 22 references still open.
- It may state two separate claims as one. 35
- Probably not a substantial claim. 19
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“scBERT [7] embeds each gene as the sum of two embeddings: one representing the gene's normalized, binned log-scale expression level, and a second which specifies the gene identity via the previously developed gene2vec [12] embedding, allowing the model to differentiate genes in the otherwise order-less data.”
From A Deep Dive into Single-Cell RNA Sequencing Foundation Models (2023) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“Even for difficult-to-distinguish CD8+ cytotoxic T cells and CD8+/CD45RA+ T cells [7], logistic regression outperformed scBERT in terms of accuracy and F1 score (Table 1).”
From A Deep Dive into Single-Cell RNA Sequencing Foundation Models (2023) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“The expression embedding - not the gene2vec embedding - drives the variability in the learned representations of a given gene for different cells and is the focus of the masking task used to pre-train the model.”
From A Deep Dive into Single-Cell RNA Sequencing Foundation Models (2023) · cites this paper as [7]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“In the hardware setup, we find the training process is accelerated by incorporating the PowerSGD (51) gradient compression with rank 32 to reduce the communication loads across different GPUs.”
From High-resolution de novo structure prediction from primary sequence (2022) · cites this paper as [51]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data > Performance on the inter-dataset data.
“The cells of the four types were randomly selected with the cell numbers 10000, 100, 10000, and 100 as reference data for finetuning.”
§ scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data > Figure 5. Model interpretability.
“It contains about 68, 000 cells within eleven subtypes of cells: CD8+ Cytotoxic T cells (30.3%), CD8+/CD45RA+ Naive Cytotoxic cells (24.3%), CD56+ NK cells (12.8%), CD4+/CD25 T Reg cells (9.0%), CD19+ B cells (8.6%), CD4+/CD45RO+ Memory cells (4.5%), CD14+ Monocyte cells (4.2%), Dendritic cells (3.1%), CD4+/CD45RA+/CD25-Naive T cells (2.7%), CD34+ cells (0.4%)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data > 5
“MacParland dataset 51 from human liver tissue contains 20 hepatic cell populations from the transcriptional profiling of 8444 cells by 10X CHROMIUM.”
§ scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data > Novel cell type detection.
“Here we adopted MacParland dataset 51 from human liver tissues with 8,434 cells belonging to 14 cell types.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Integrated analysis of multimodal single-cell data
Hao, Y., Hao, S., Andersen-Nissen, E., Mauck, W. M., Zheng, S., Butler, A., Lee, M. J., Wilk, A. J., Darby, C., Zagar, M., et al. (2020) · doi:10.1101/2020.10.12.335331 · Cited by 15,508
Double-slit photoelectron interference in strong-field ionization of the neon dimer
Kunitski, M., Eicke, N., Huber, P., Köhler, J., Zeller, S., Voigtsberger, J., Schlott, N., Henrichs, K., Sann, H., Trinter, F., et al. (2019) · Nature Communications · doi:10.1038/s41467-018-07882-8 · Cited by 8,162
Double-slit photoelectron interference in strong-field ionization of the neon dimer
Kunitski, M., Eicke, N., Huber, P., Köhler, J., Zeller, S., Voigtsberger, J., Schlott, N., Henrichs, K., Sann, H., Trinter, F., et al. (2019) · Institutional Repository of Leibniz Universität Hannover (Leibniz Universität Hannover) · doi:10.15488/5174 · Cited by 8,161
Massively parallel digital transcriptional profiling of single cells
Zheng, G. X. Y., Terry, J. M., Belgrader, P., Ryvkin, P., Bent, Z. W., Wilson, R., Ziraldo, S. B., Wheeler, T. D., McDermott, G. P., Zhu, J., et al. (2017) · Nature Communications · doi:10.1038/ncomms14049 · Cited by 7,353
U1 snRNP regulates cancer cell migration and invasion in vitro
Oh, J.-M., Venters, C. C., Di, C., Pinto, A. M., Wan, L., Younis, I., Cai, Z., Arai, C., So, B. R., Duan, J., et al. (2020) · Nature Communications · doi:10.1038/s41467-019-13993-7 · Cited by 7,157
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Anomalous collapses of Nares Strait ice arches leads to enhanced export of Arctic sea ice
Moore, G. W. K., Howell, S. E. L., Brady, M., Xu, X., & McNeil, K. (2021) · Nature Communications · doi:10.1038/s41467-020-20314-w · Cited by 5,155
Aran, D., Looney, A. P., Liu, L., Wu, E., Fong, V., Hsu, A., Chak, S., Naikawadi, R. P., Wolters, P. J., Abate, A. R., et al. (2019) · Nature Immunology · doi:10.1038/s41590-018-0276-y · Cited by 4,984
Research works printed but not linked: 7
- 10.1038/s41467-020-16839-9 · DOI · Paper page
- 10.1038/s41467-020-16839-9 · DOI · Paper page
- 10.1186/s13059-019-1727-y · DOI · Paper page
- 10.1038/s41467-020-16839-9 · DOI · Paper page
- 10.1186/s13059-019-1727-y · DOI · Paper page
- Do Transformers Really Perform Badly for Graph Representation? https://proceedings.neurips.cc/paper/2021/hash/f1c1592588411002af340cbaedd6fc33-Abstract.html.
- 10.1186/s13059-020-1935-5 · DOI · Paper page
Cited by
scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., & Wang, B. (2023) · doi:10.1101/2023.04.30.538439 · Cited by 1,694
Large-scale foundation model on single-cell transcriptomics
Hao, M., Gong, J., Zeng, X., Liu, C., Guo, Y., Cheng, X., Wang, T., Ma, J., Zhang, X., & Song, L. (2024) · Nature Methods · doi:10.1038/s41592-024-02305-7 · Cited by 628
High-resolution de novo structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392
Profiling cell identity and tissue architecture with single-cell and spatial transcriptomics
Gulati, G. S., D’Silva, J. P., Liu, Y., Wang, L., & Newman, A. M. (2025) · Nature Reviews Molecular Cell Biology · doi:10.1038/s41580-024-00768-2 · Cited by 293
A Review of Single-Cell RNA-Seq Annotation, Integration, and Cell–Cell Communication
Cheng, C., Chen, W., Jin, H., & Chen, X. (2023) · Cells · doi:10.3390/cells12151970 · Cited by 224
Metadata
- Authors
- Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., & Yao, J.
- Affiliations
- Tencent (China) (CN)
- Shanghai Jiao Tong University (CN)
- Harvard University (US)
- The University of Texas at Arlington (US)
- Corresponding author
- Lu, Hui
- Yao, Jianhua
- Venue
- Nature Machine Intelligence
- Publisher
- openRxiv
- Published
- 2021-12-07
- Volume
- 4
- Issue
- 10
- Pages
- 852-866
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W4297243391
- Cited by
- 617
- References
- 68
- Citations by year
- 2026: 172 · 2025: 233 · 2024: 158 · 2023: 52 · 2022: 2
- FWCI
- 36.41
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Nature Machine Intelligence (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
- Funders
- Yale University
- Shanghai Jiao Tong University
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
- Keywords
- cell type annotation, single-cell RNA sequencing, transformer models, self-supervised pre-training, gene-gene interaction, marker genes, fine-tuning, model interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data. https://app.cochat.ai/papers/10.1101/2021.12.05.471261
- The paper
- Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., & Yao, J. (2021). scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data. Nature Machine Intelligence. https://doi.org/10.1101/2021.12.05.471261
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.