scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data

May combine two claims.

With only 30% cells from Zheng68K, scBERT beat all comparison methods, and its performance improved rapidly as reference cell number increased.

“With only 30% cells from Zheng 68K dataset), scBERT beated all comparison moethods, and its performance improved rapidly as reference cell number increased.”

May combine two claims.

The list of top-tier methods changes across different tasks and datasets while scBERT is always among it.

“Notably, the list of the top-tier methods changes across different tasks and datasets while scBERT is always among it.”

May combine two claims.

Compared to SciBet and scmap_cluster, scBERT achieves superior accuracy on both novel class (scBERT=0.329 vs SciBet=0.174 and scmap_cluster=0.174) and known classes (scBERT=0.942 vs SciBet=0.784 and scmap_cluster=0.666).

“Compared to SciBet and scmap_cluster, our method achieves the superior accuracy on both the novel class (scBERT=0.329vs. SciBet=0.174and scmap_cluster=0.174)and known classes (scBERT=0.942vs. SciBet=0.784and scmap_cluster=0.666).”

May combine two claims.

scBERT correctly annotated most cells (>97%) in the Muraro dataset and over 99% in the other three datasets.

“scBERT correctly annotated most cells (> 97%) in the Muraro dataset and over 99% in the other three datasets, demonstrating the superb and stable performance of our method in cross-cohort tasks.”

May combine two claims.

scNym misclassified the alpha cells as beta cell type and was confused with the beta and delta cells.

“In the contrast, scNym misclassified the alpha cells as beta cell type and was confused with the beta and delta cells (Fig. 3e-f).”

May combine two claims.

scBERT could easily deal with large-scale datasets at both the pre-training and fine-tuning stages, benefiting from mini-batch sampling and the efficient Performer encoder.

“Conversely, benefiting from mini-batch sampling and the efficient Performer encoder, scBERT could easily deal with large-scale datasets at both the pre-training and the fine-tuning stage.”

May combine two claims.

scBERT performed the best on novel cell types and achieved first-class performance on known cell types.

“Not surprisingly, scBERT performed the best on novel cell types and achieved the first-class performance on the known cell types (Fig. 4).”

May combine two claims.

ML-based methods tend to learn high-level implicit cell-type-specific patterns, whereas non-ML based methods usually simply find differentially expressed genes using statistics analysis.

“we could observe that ML-based methods tend to learn high-level implicit cell-type-specific patterns (i.e.discovering some genes with high rank across cell types), whereas non-ML based methods usually simply find differentially expressed genes using statistics analysis.”

May combine two claims.

With deletion of 100% marker genes, scBERT can still efficiently learn informative gene patterns and achieve performance on par with the best performance achieved by comparison methods with all marker genes on the representative Zheng68K dataset.

“With deletion of 100% marker genes, scBERT can still efficiently learn the informative gene patterns and achieve the performance on par with the best performance achieved by comparison methods with all marker genes on the representative Zheng68K dataset (Extended Data Figure 1b).”

May combine two claims.

CellID_cell performed well on known cell types, while failed to discover any novel cells.

“CellID_cell performed well on known cell types, while failed to discover any novel cells.”

May combine two claims.

For cross-cohort data, scBERT achieved superior performance with accuracy of 0.992 compared to scNym (accuracy=0.904) with a large margin (about 10%) and outperformed other popular methods (accuracy: SciBet 0.985, Seurat 0.984, SingleR 0.987).

“For cross-cohort data, scBERT achieved a superior performance with accuracy of 0.992 compared to scNym (accuracy=0.904)with a large margin (about 10%)and outperformed other popular methods (accuracy: SciBet 0.985, Seurat 0.984, SingleR 0.987) (Fig. 3b).”

May combine two claims.

On the Muraro dataset, the ARI score for the UMAP based on scBERT embedding is 0.95 and for the UMAP based on raw expression is 0.87.

“Left UMAP is based on the scBERT embedding of each cell, while right UMAP is based on the raw expression of each cell.”

• Also stated in: § Figure 5. Model interpretability. · plain

  “ARI=0.95ARI=0.87”

May combine two claims.

scBERT embedding exhibits cell-type-specific representation, with an example representation of a gene significantly enriched in alpha cell type.

“We observed that the scBERT embedding exhibits cell-type-specific representation (the example representation of gene is significantly enriched in alpha cell type), which is naturally suitable for down-streaming cell type annotation task.”

May combine two claims.

The model performs stably with 4 and 6 Performer encoder layers but might suffer from under-fitting or over-fitting when decreasing or increasing the number of layers.

“Similarly, the model performs stable with 4 and 6 of Performer encoder layers but might suffer from under-fitting or over-fitting problem when decreasing or increasing the number of layers.”

May combine two claims.

Small fluctuations of the hyperparameters had little effect on model performance, verifying the robustness of scBERT.

“Overall, the small fluctuations of the above parameters had little effect on the performance of the model, which also verified the robustness of scBERT.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Enrichment analysis of top-50-attention gene lists revealed interesting relationships between top-enriched terms and the corresponding cell types.

“Enrichment analysis was performed for the top-50-attention gene lists using various gene-set libraries and the results revealed there were some interesting relationships between top-enriched terms and the corresponding cell types.”

• AssumptionRests on an assumption: Attention weights reflect the contribution of each gene and the interaction of gene pairs.

May combine two claims.

scBERT can generate different representations of the same gene for different cell inputs, while gene2vec generates the same representation of the same gene for different cell inputs.

“Our model could generate different representations of the same gene for different cell inputs, while gene2vec generated all the same representation of the same gene for different cell inputs.”

May combine two claims.

Gene2vec provided an embedding of 200 dimensions and achieved the best performance compared to other dimensions.

“Second, the gene2vec provided an embedding of 200 dimensions and achieved the best performance compared to other dimensions.”

May combine two claims.

It is still challenging to distinguish true biological diversity from technical differences and preserve important biological variations despite batch-effect correction methods.

“Although there exist batch-effect correction methods, it's still challenging to distinguish true biological diversity from technical differences and thus preserve important biological variations 13 .”

• cites [13] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

These uncertainties in performance reflect limitations of comparison methods in generalizability, as well as generalization of our method across all benchmarking datasets.

“These uncertainties in performance reflect the limitations of the comparison methods in their generalizability, as well as the generalization of our method across all the benchmarking datasets.”

May combine two claims.

Pretraining enables scBERT to learn general syntax of gene-gene interactions, which helps to remove batch effects across datasets and improve generalizability.

“Pretraining enables the model to learn general syntax of gene-gene interactions, which helps to remove the batch effects across datasets and improve the generalizability (Extended Data Figure 1a).”

• AssumptionRests on an assumption: BERT's paradigm of pre-training and fine-tuning enables the use of large-scale unlabeled data to improve the generalizability of AI model.

Classified as probably not a substantial claim.

BERT has made breakthrough progress in NLP due to the powerful self-attention mechanism and long-range information integration capability introduced by Transformer layers.

“It has made breakthrough progress in the fields of natural language processing (NLP) due to the powerful self-attention mechanism and long-range information integration capability introduced by Transformer layers 24,25 .”

• cites [24] Parmar, N. et al. Image Transformer. 4055–4064 Preprint at https://proceedings.mlr.press/v80/parmar18a.html (2018).Not checked yet

  Not checked in the free run.

• cites [25] 2018, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the ConferenceNot checked yet

  Not checked in the free run.

May combine two claims.

Existing marker-gene-based cell type annotation methods depend on manually curated marker genes and are prone to bias and errors.

“However, selecting the marker genes depends on researchers' prior knowledge, and is therefore prone to bias and errors 7 .”

• cites [7] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Fine-tuning ensures that the output embedding for each gene encodes context information more relevant to the transcriptional profiles of the reference dataset.

“Fine-tuning ensures that the output embedding for each gene encodes context information that is more relevant to the transcriptional profiles of the reference dataset.”

Classified as probably not a substantial claim.

No manual selection of marker genes was included for an unbiased and fair comparison of all methods.

“No manual selection of the marker genes was included for an unbiased and fair comparison of all the methods.”

May combine two claims.

scBERT can correctly discover novel cell types that are not present in original reference datasets and remain accurate prediction performance of other cell types.

“Taken together, these results suggested that scBERT can correctly discover novel cell types that are not present in original reference datasets and remain accurate prediction performance of other cell types.”

• AssumptionRests on an assumption: The reference dataset may not cover all cell types present in the query dataset in most tasks.

May combine two claims.

scBERT showed robustness in identifying cells from different sequencing technologies, experiments, different disease states (T2D and health), and even different organs.

“scBERT showed its robustness in identifying cells from different sequencing technologies, experiments, different disease states (type 2 diabetes (T2D) and health), and even different organs.”

Classified as probably not a substantial claim. May combine two claims.

The frequency of word occurrence in text is valuable information and is often transformed as bag-of-words by term frequency statistic analysis for downstream NLP tasks.

“It's worth noting that the frequency of the word occurrence in a text is valuable information for text analysis and is often 320 transformed as bag-of-words by term frequency statistic analysis for downstream tasks in the area of 321 NLP 48 .”

• cites [48] 2010, Understanding bag-of-words model: a statistical framework, International Journal of Machine Learning and Cybernetics 2010 1:1Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Masking only the non-zero values may lower the utilization of the single-cell data for pre-training, due to their minority.

“However, masking only the non-zero values may lower the utilization of the single-cell data for pre-training, due to their minority.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

scBERT can also be migrated to other tasks by simply modifying output and supervision signals.

“Beyond that, scBERT can also be migrated to other tasks by simply modifying output and supervision signals.”

May combine two claims.

scBERT extracts useful attention patterns on gene expressions and interactions from large-scale various scRNA-seq data, alleviating fine-tuning efforts on specific downstream tasks.

“The scBERT model extracts the useful attention pattern on gene expressions and interactions from a large scale of various scRNA-seq data, alleviating the efforts of the fine-tuning process on the specific downstream tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Word embedding technology helps better represent text by ensuring words with similar meanings have similar representations.

“The word embedding technology helps to better represent the text by assuring the words with similar meanings have a similar representation 47 .”

• cites [47] Goldberg, Y. Neural Network Methods for Natural Language Processing. http://dx.doi.org/10.2200/S00762ED1V01Y201703HLT037 10, 1–311 (2017). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

scBERT has superb and stable performance in cross-cohort tasks.

“demonstrating the superb and stable performance of our method in cross-cohort tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Well-documented marker lists associated with well-defined cell types from CellMarker were used to avoid bias introduced by marker selection.

“To avoid bias introduced by marker selection, welldocumented marker lists associated with well-defined cell types from CellMarker 56 were used.”

• cites [56] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Accuracy, macro-F1 score, and confusion matrix were employed as evaluation metrics for benchmarking performance for fair comparison.

“Notably, we employed the accuracy, macro-F1 score, and confusion matrix as evaluation metrics for benchmarking the performance of cell type annotation methods on their classification ability for a fair comparison in this study.”

May combine two claims.

scBERT is efficient in learning single-cell-specific representation, which can be used for downstream analysis.

“indicating the efficiency of scBERT in learning single-cell-specific representation, which can be used for downstream analysis (Fig. 5d).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Gene interactions usually exist in the form of networks, such as gene regulatory networks and biological signaling pathways.

“Second, gene interactions usually exist in the form of networks (i.e., gene regulatory networks and biological signaling pathways) 43 , and this kind of prior knowledge hasn't been explicitly incorporated in scBERT.”

• cites [43] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Gene2vec is based on bulk data and measures the average expression of genes from tissues, which is the sum of cell type-specific gene expression weighted by cell type proportions.

“Gene2vec is based on bulk data 28 , which measures the average expression of genes from tissues and is the sum of cell type-specific gene expression weighted by cell type proportions 60 .”

• cites [28] 2019, Gene2vec: distributed representation of genes based on co-expression, BMC Genomics 2019 20:1Not checked yet

  Not checked in the free run.

• cites [60] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

SCD5 has not been reported as a cell-type-specific marker for beta cells, but in a GWAS study, a novel loci for T2D susceptibility was fine-mapped to a coding variant of SCD.

“For instance, SCD5 hasn't been reported as a cell-type-specific marker for beta cells, but in a GWAS study, a novel loci for T2D susceptibility was fine-mapped to a coding variant of SCD 42 .”

• cites [42] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Extensive and rigorous benchmark studies validated the superior performance of scBERT on cell type annotation, novel cell type discovery, robustness to batch effect, and model interpretability.

“Extensive and rigorous benchmark studies validated the superior performance of scBERT on cell type annotation, novel cell type discovery, robustness to batch effect, and model interpretability.”

May combine two claims.

scBERT surpasses existing advanced methods on diverse benchmarks collectively involving 9 single-cell datasets, 17 major organs/tissues, more than 50 cell types, over 500,000 cells, and mainstream single-cell omics technologies, indicating its generalization and robustness.

“scBERT surpasses the existing advanced methods on diverse benchmarks collectively involving 9 singlecell datasets, 17 major organs/tissues, more than 50 cell types, over 500,000 cells, and the mainstream single-cell omics technologies (i.e., Drop-seq, 10X, SMART-seq, and Sanger-Nuclei), indicating its generalization and robustness.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Only scRNA-seq datasets that provided highly credible cell type annotation and had been cited by the majority of cell type annotation methods for performance evaluation were included.

“To this end, we only included scRNA-seq datasets that provided highly credible cell type annotation and had been cited by the majority of the cell type annotation methods for performance evaluation.”

May combine two claims.

Transformer with global receptive field can effectively learn global representation and long-range dependency without absolute position information, achieving excellent performance on non-sequential data such as images and tables.

“Transformer with global receptive field could effectively learn global representation and longrange dependency without absolute position information, achieving excellent performance on nonsequential data (i.e., images, tables) 24,27 .”

• cites [24] Parmar, N. et al. Image Transformer. 4055–4064 Preprint at https://proceedings.mlr.press/v80/parmar18a.html (2018).Not checked yet

  Not checked in the free run.

• cites [27] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

For each dataset, 5-fold cross-validation was applied to avoid the influence of random results on the conclusion.

“For each of the datasets, we applied the 5-fold cross-validation strategy to avoid the influence of random results on the conclusion.”

May combine two claims.

scBERT is prone to generate an efficient encoder and provide a general embedding that better represents the gene expression of each cell by revealing critical patterns with lower data noise.

“Following the BERT's pre-training and fine-tuning paradigm, our method is prone to generate an efficient encoder and provide a general embedding that better represents the gene expression of each cell by revealing critical patterns with lower data noise.”

Probably not a substantial claim: may not pass S3 (load-bearing).

An over-parameterized model with 20 attention heads faces a risk of over-fitting, especially when applied to small datasets.

“However, the over-parameterized model (with 20 attention heads) faces a risk of over-fitting, especially when applying to small datasets.”

Classified as probably not a substantial claim.

The performance of cell type annotation methods was measured by accuracy and F1 score on n=9 datasets using 5-fold cross-validation.

“Performance of cell type annotation methods measured by accuracy and F1 score on n=9 datasets using 5-fold cross-validation.”

Classified as probably not a substantial claim.

A strict evaluation method for novel cell types was employed, with accuracy calculated on the union set of cells with the novel cell type label and cells predicted as novel cell types.

“Notably, we employed a strict evaluation method for novel cell types with the accuracy calculated on the union set of cells with the novel cell type label and the cells that are predicted as novel cell types.”

May combine two claims.

Gene2vec maintains general coexpression patterns of genes but stays away from strong noise and high sparsity of single-cell sequencing.

“In this regard, gene2vec maintains the general coexpression patterns of genes but stays away from strong noise and high sparsity of single-cell sequencing.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Zheng68k dataset contains about 68,000 cells within eleven subtypes.

“It contains about 68, 000 cells within eleven subtypes of cells”

• cites [35] 2017, Massively parallel digital transcriptional profiling of single cells, Nature Communications 2017 8:1Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Cross-validation was employed in all benchmarking experiments.

“Cross-validation was employed in all the benchmarking experiments and standard deviations were drawn in the figures.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Zheng68k dataset is a classic PBMC dataset by 10X CHROMIUM widely used for cell type annotation performance assessment.

“The Zheng68k is a classic PBMC dataset by 10X CHROMIUM that is widely used for cell type annotation performance acessment35 .”

• cites [35] 2017, Massively parallel digital transcriptional profiling of single cells, Nature Communications 2017 8:1Not checked yet

  Not checked in the free run.