CellPLM: Pre-training of Cell Language Model Beyond Single Cells
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
The current state-of-the-art single-cell pre-trained models are greatly inspired by the success of large language models. They trained transformers by treating genes as tokens and cells as sentences. However, three fundamental differences between single-cell data and natural language data are overlooked: (1) scRNA-seq data are presented as bag-of-genes instead of sequences of RNAs; (2) Cell-cell relations are more intricate and important than inter-sentence relations; and (3) The quantity of single-cell data is considerably inferior to text data, and they are very noisy. In light of these characteristics, we propose a new pre-trained model CellPLM , which takes cells as tokens and tissues as sentences. In addition, we leverage spatially-resolved transcriptomic data in pre-training to facilitate learning cell-cell relationships and introduce a Gaussian mixture prior distribution as an additional inductive bias to overcome data limitation. CellPLM is the first single-cell pre-trained transformer that encodes cell-cell relations and it consistently outperforms existing pre-trained and non-pre-trained models in diverse downstream tasks, with 100x times higher inference speed compared to existing pre-trained models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Inference and analysis of cell-cell communication using CellChat
Jin, S., Guerrero-Juarez, C. F., Zhang, L., Chang, I., Ramos, R., Kuan, C.-H., Myung, P., Plikus, M. V., & Nie, Q. (2021) · Nature Communications · doi:10.1038/s41467-021-21246-9 · Cited by 8,409
Cellpose: a generalist algorithm for cellular segmentation
Stringer, C., Wang, T., Michaelos, M., & Pachitariu, M. (2021) · Nature Methods · doi:10.1038/s41592-020-01018-x · Cited by 4,369
Visualization and analysis of gene expression in tissue sections by spatial transcriptomics
Ståhl, P. L., Salmén, F., Vickovic, S., Lundmark, A., Navarro, J. F., Magnusson, J., Giacomello, S., Asp, M., Westholm, J. O., Huss, M., et al. (2016) · Science · doi:10.1126/science.aaf2403 · Cited by 3,834
Performance of neural network basecalling tools for Oxford Nanopore sequencing
Wick, R. R., Judd, L. M., & Holt, K. E. (2019) · Genome Biology · doi:10.1186/s13059-019-1727-y · Cited by 3,221
Slingshot: cell lineage and pseudotime inference for single-cell transcriptomics
Street, K., Risso, D., Fletcher, R. B., Das, D., Ngai, J., Yosef, N., Purdom, E., & Dudoit, S. (2018) · BMC Genomics · doi:10.1186/s12864-018-4772-0 · Cited by 3,159
Deep generative modeling for single-cell transcriptomics
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., & Yosef, N. (2018) · Nature Methods · doi:10.1038/s41592-018-0229-2 · Cited by 2,530 · cited 2×
“By feeding the batch label to the decoder, a batch-effect-free latent space can be achieved, as empirically evidenced in scVI (Lopez et al., 2018).”
BackgroundOpportunities and challenges in long-read sequencing data analysis
Amarasinghe, S. L., Su, S., Dong, X., Zappia, L., Ritchie, M. E., & Gouil, Q. (2020) · Genome Biology · doi:10.1186/s13059-020-1935-5 · Cited by 2,530
Regev, A., Teichmann, S. A., Lander, E. S., Amit, I., Benoist, C., Birney, E., Bodenmiller, B., Campbell, P., Carninci, P., Clatworthy, M., et al. (2017) · eLife · doi:10.7554/elife.27041 · Cited by 2,232 · cited 1×
“For comparison, the high-quality filtered English dataset extracted from Common Crawl corpora (Wenzek et al., 2020) consists of 32 billion sentences, whereas the largest collection of single-cell datasets, namely the Human Cell Atlas (Regev et al., 2017), includes less than 50 million cells.”
BackgroundDixit, A., Parnas, O., Li, B., Chen, J., Fulco, C. P., Jerby-Arnon, L., Marjanovic, N. D., Dionne, D., Burks, T., Raychowdhury, R., et al. (2016) · Cell · doi:10.1016/j.cell.2016.11.038 · Cited by 1,863
Recovering Gene Interactions from Single-Cell Data Using Data Diffusion
van Dijk, D., Sharma, R., Nainys, J., Yim, K., Kathail, P., Carr, A. J., Burdziak, C., Moon, K. R., Chaffer, C. L., Pattabiraman, D., et al. (2018) · Cell · doi:10.1016/j.cell.2018.05.061 · Cited by 1,829
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Deciphering cell–cell interactions and communication from gene expression
Armingol, E., Officer, A., Harismendy, O., & Lewis, N. E. (2021) · Nature Reviews Genetics · doi:10.1038/s41576-020-00292-x · Cited by 1,267 · cited 2×
“The other type of relations, cell-cell communications, plays an essential role in determining cell development and cell states (Armingol et al., 2021).”
BackgroundAdamson, B., Norman, T. M., Jost, M., Cho, M. Y., Nuñez, J. K., Chen, Y., Villalta, J. E., Gilbert, L. A., Horlbeck, M. A., Hein, M. Y., et al. (2016) · Cell · doi:10.1016/j.cell.2016.11.048 · Cited by 1,193
Transfer learning enables predictions in network biology
Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., Al Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng, Z., Liu, X. S., et al. (2023) · Nature · doi:10.1038/s41586-023-06139-9 · Cited by 1,193 · cited 8×
“In addition to CellPLM , we include three baselines, i.e., PCA, Geneformer (Theodoris et al., 2023) and scGPT(Cui et al., 2023).”
BackgroundMethodInfluentialSingle-cell RNA-seq denoising using a deep count autoencoder
Eraslan, G., Simon, L. M., Mircea, M., Mueller, N. S., & Theis, F. J. (2019) · Nature Communications · doi:10.1038/s41467-018-07931-2 · Cited by 1,084 · cited 3×
“2019) introduced an autoencoder framework based on zero inflated negative binomial (ZINB) distribution.”
MethodDeep learning and alignment of spatially resolved single-cell transcriptomes with Tangram
Biancalani, T., Scalia, G., Buffoni, L., Avasthi, R., Lu, Z., Sanger, A., Tokcan, N., Vanderburg, C. R., Segerstolpe, Å., Zhang, M., et al. (2021) · Nature Methods · doi:10.1038/s41592-021-01264-7 · Cited by 931 · cited 1×
“Following the setting of baselines including SpaGE (Abdelaal et al., 2020), stPlus (Shengquan et al., 2021), gimVI (Lopez et al., 2019) and Tangram (Biancalani et al., 2021), we impute the unmeasured genes of the SRT dataset utilizing a scRNA-seq dataset as reference.”
MethodValidation of noise models for single-cell transcriptomics
Grün, D., Kester, L., & van Oudenaarden, A. (2014) · Nature Methods · doi:10.1038/nmeth.2930 · Cited by 721
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 1×
“The first such pre-trained model for single-cell data, scBERT (Yang et al., 2022), takes genes as tokens and leverages an efficient transformer (Choromanski et al., 2020) to encode over 16,000 gene tokens for each cell.”
MethodXu, C., Lopez, R., Mehlman, E., Regier, J., Jordan, M. I., & Yosef, N. (2021) · Molecular Systems Biology · doi:10.15252/msb.20209620 · Cited by 691 · cited 1×
“Prior studies have employed variational autoencoders for single-cell analysis, which typically assumes an isotropic Gaussian distribution as the prior distribution of the latent space (Lopez et al., 2018; Xu et al., 2021).”
BackgroundscGen predicts single-cell perturbation responses
Lotfollahi, M., Wolf, F. A., & Theis, F. J. (2019) · Nature Methods · doi:10.1038/s41592-019-0494-8 · Cited by 661
Neuronal vulnerability and multilineage diversity in multiple sclerosis
Schirmer, L., Velmeshev, D., Holmqvist, S., Kaufmann, M., Werneburg, S., Jung, D., Vistnes, S., Stockley, J. H., Young, A., Steindel, M., et al. (2019) · Nature · doi:10.1038/s41586-019-1404-z · Cited by 638
Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., & Yao, J. (2021) · doi:10.1101/2021.12.05.471261 · Cited by 617
Power analysis of single-cell RNA-sequencing experiments
Svensson, V., Natarajan, K. N., Ly, L.-H., Miragaia, R. J., Labalette, C., Macaulay, I. C., Cvejic, A., & Teichmann, S. A. (2017) · Nature Methods · doi:10.1038/nmeth.4220 · Cited by 568 · cited 2×
“To make things worse, single-cell data often suffer from technical artifacts and dropout events (Svensson et al., 2017; Qiu, 2020), as well as significant batch effects between sequencing platforms and experiments (Tran et al., 2020; Argelaguet et al., 2021).”
BackgroundComputational principles and challenges in single-cell data integration
Argelaguet, R., Cuomo, A. S. E., Stegle, O., & Marioni, J. C. (2021) · Nature Biotechnology · doi:10.1038/s41587-021-00895-7 · Cited by 472
Exploring genetic interaction manifolds constructed from rich single-cell phenotypes
Norman, T. M., Horlbeck, M. A., Replogle, J. M., Ge, A. Y., Xu, A., Jost, M., Gilbert, L. A., & Weissman, J. S. (2019) · Science · doi:10.1126/science.aax4438 · Cited by 392
Li, Y., Ren, P., Dawson, A., Vasquez, H. G., Ageedi, W., Zhang, C., Luo, W., Chen, R., Li, Y., Kim, S., et al. (2020) · Circulation · doi:10.1161/circulationaha.120.046528 · Cited by 344 · cited 1×
“To evaluate the transferability of the pre-trained model, we extract cell embeddings from the pre-trained encoder on a public dataset from Li et al. (2020), which is not included in pre-train data.”
MethodPredicting cell-to-cell communication networks using NATMI
Hou, R., Denisenko, E., Ong, H. T., Ramilowski, J. A., & Forrest, A. R. R. (2020) · Nature Communications · doi:10.1038/s41467-020-18873-z · Cited by 285
ACTINN: automated identification of cell types in single cell RNA sequencing
Ma, F., & Pellegrini, M. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz592 · Cited by 252
SpaGE: Spatial Gene Enhancement using scRNA-seq
Abdelaal, T., Mourragui, S., Mahfouz, A., & Reinders, M. J. T. (2020) · Nucleic Acids Research · doi:10.1093/nar/gkaa740 · Cited by 207 · cited 1×
“Following the setting of baselines including SpaGE (Abdelaal et al., 2020), stPlus (Shengquan et al., 2021), gimVI (Lopez et al., 2019) and Tangram (Biancalani et al., 2021), we impute the unmeasured genes of the SRT dataset utilizing a scRNA-seq dataset as reference.”
MethodTransformer for one stop interpretable cell type annotation
Chen, J., Xu, H., Tao, W., Chen, Z., Zhao, Y., & Han, J.-D. J. (2023) · Nature Communications · doi:10.1038/s41467-023-35923-4 · Cited by 189 · cited 13×
“In addition to CellPLM , we include three baselines, i.e., PCA, Geneformer (Theodoris et al., 2023) and scGPT(Cui et al., 2023).”
BackgroundMethodInfluential
Cited by
Nicheformer: a foundation model for single-cell and spatial omics
Tejada-Lapuerta, A., Schaar, A. C., Gutgesell, R., Palla, G., Halle, L., Minaeva, M., Vornholz, L., Dony, L., Drummer, F., Richter, T., et al. (2025) · Nature Methods · doi:10.1038/s41592-025-02814-z · Cited by 100
Evaluating the Utilities of Foundation Models in Single-cell Data Analysis
Liu, T., Li, K., Wang, Y., Li, H., & Zhao, H. (2023) · doi:10.1101/2023.09.08.555192 · Cited by 62
“Some single-cell FMs focus on cell-type annotation or gene function prediction, including scBERT [29], CellLM [30], and Geneformer [31], while others aim to create a foundation model in this area that can handle multiple tasks, including scGPT [32], scFoundation [33], tGPT [34], GeneCompass [35],…”
Single-cell foundation models: bringing artificial intelligence into cell biology
Baek, S., Song, K., & Lee, I. (2025) · Experimental & Molecular Medicine · doi:10.1038/s12276-025-01547-5 · Cited by 46
“Moreover, we highlight several noteworthy models — scBERT 13 , geneformer 14,18 , tGPT 29 , CellLM 15 , CellPLM 48 , scGPT 7,10 , scFoundation 34 , scPRINT 19 , UCE 16 , GeneCompass 17 , Nicheformer 11 and SCimilarity 38 .”
PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction
Wenteler, A., Occhetta, M., Branson, N., Curean, V., Huebner, M., Dee, W., Connell, W., Chung, S. P., Hawkins-Hooker, A., Ektefaie, Y., et al. (2024) · doi:10.1101/2024.10.02.616248 · Cited by 37
Evaluating the learnability of single-cell large language models on multiple tasks
Yan, Y., Wang, X., & Song, D. (2026) · doi:10.21203/rs.3.rs-8919408/v1 · Cited by 1
Metadata
- Authors
- Wen, H., Tang, W., Dai, X., Ding, J., Jin, W., Xie, Y., & Tang, J.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2023-10-05
- OpenAlex
- W4387358535
- Cited by
- 71
- References
- 31
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.