CellPLM: Pre-training of Cell Language Model Beyond Single Cells
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
The current state-of-the-art single-cell pre-trained models are greatly inspired by the success of large language models. They trained transformers by treating genes as tokens and cells as sentences. However, three fundamental differences between single-cell data and natural language data are overlooked: (1) scRNA-seq data are presented as bag-of-genes instead of sequences of RNAs; (2) Cell-cell relations are more intricate and important than inter-sentence relations; and (3) The quantity of single-cell data is considerably inferior to text data, and they are very noisy. In light of these characteristics, we propose a new pre-trained model CellPLM , which takes cells as tokens and tissues as sentences. In addition, we leverage spatially-resolved transcriptomic data in pre-training to facilitate learning cell-cell relationships and introduce a Gaussian mixture prior distribution as an additional inductive bias to overcome data limitation. CellPLM is the first single-cell pre-trained transformer that encodes cell-cell relations and it consistently outperforms existing pre-trained and non-pre-trained models in diverse downstream tasks, with 100x times higher inference speed compared to existing pre-trained models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Cellpose: a generalist algorithm for cellular segmentation
Stringer, C., Wang, T., Michaelos, M., & Pachitariu, M. (2021) · Nature Methods · doi:10.1038/s41592-020-01018-x · Cited by 4,369
Visualization and analysis of gene expression in tissue sections by spatial transcriptomics
Ståhl, P. L., Salmén, F., Vickovic, S., Lundmark, A., Navarro, J. F., Magnusson, J., Giacomello, S., Asp, M., Westholm, J. O., Huss, M., et al. (2016) · Science · doi:10.1126/science.aaf2403 · Cited by 3,834
Performance of neural network basecalling tools for Oxford Nanopore sequencing
Wick, R. R., Judd, L. M., & Holt, K. E. (2019) · Genome Biology · doi:10.1186/s13059-019-1727-y · Cited by 3,221
Deep generative modeling for single-cell transcriptomics
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., & Yosef, N. (2018) · Nature Methods · doi:10.1038/s41592-018-0229-2 · Cited by 2,530 · cited 2×
“By feeding the batch label to the decoder, a batch-effect-free latent space can be achieved, as empirically evidenced in scVI (Lopez et al., 2018).”
BackgroundOpportunities and challenges in long-read sequencing data analysis
Amarasinghe, S. L., Su, S., Dong, X., Zappia, L., Ritchie, M. E., & Gouil, Q. (2020) · Genome Biology · doi:10.1186/s13059-020-1935-5 · Cited by 2,530
Regev, A., Teichmann, S. A., Lander, E. S., Amit, I., Benoist, C., Birney, E., Bodenmiller, B., Campbell, P., Carninci, P., Clatworthy, M., et al. (2017) · eLife · doi:10.7554/elife.27041 · Cited by 2,232 · cited 1×
“For comparison, the high-quality filtered English dataset extracted from Common Crawl corpora (Wenzek et al., 2020) consists of 32 billion sentences, whereas the largest collection of single-cell datasets, namely the Human Cell Atlas (Regev et al., 2017), includes less than 50 million cells.”
BackgroundDixit, A., Parnas, O., Li, B., Chen, J., Fulco, C. P., Jerby-Arnon, L., Marjanovic, N. D., Dionne, D., Burks, T., Raychowdhury, R., et al. (2016) · Cell · doi:10.1016/j.cell.2016.11.038 · Cited by 1,863
Recovering Gene Interactions from Single-Cell Data Using Data Diffusion
van Dijk, D., Sharma, R., Nainys, J., Yim, K., Kathail, P., Carr, A. J., Burdziak, C., Moon, K. R., Chaffer, C. L., Pattabiraman, D., et al. (2018) · Cell · doi:10.1016/j.cell.2018.05.061 · Cited by 1,829
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Cited by
Nicheformer: a foundation model for single-cell and spatial omics
Tejada-Lapuerta, A., Schaar, A. C., Gutgesell, R., Palla, G., Halle, L., Minaeva, M., Vornholz, L., Dony, L., Drummer, F., Richter, T., et al. (2025) · Nature Methods · doi:10.1038/s41592-025-02814-z · Cited by 100
Evaluating the Utilities of Foundation Models in Single-cell Data Analysis
Liu, T., Li, K., Wang, Y., Li, H., & Zhao, H. (2023) · doi:10.1101/2023.09.08.555192 · Cited by 62
“Some single-cell FMs focus on cell-type annotation or gene function prediction, including scBERT [29], CellLM [30], and Geneformer [31], while others aim to create a foundation model in this area that can handle multiple tasks, including scGPT [32], scFoundation [33], tGPT [34], GeneCompass [35],…”
Single-cell foundation models: bringing artificial intelligence into cell biology
Baek, S., Song, K., & Lee, I. (2025) · Experimental & Molecular Medicine · doi:10.1038/s12276-025-01547-5 · Cited by 46
“Moreover, we highlight several noteworthy models — scBERT 13 , geneformer 14,18 , tGPT 29 , CellLM 15 , CellPLM 48 , scGPT 7,10 , scFoundation 34 , scPRINT 19 , UCE 16 , GeneCompass 17 , Nicheformer 11 and SCimilarity 38 .”
PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction
Wenteler, A., Occhetta, M., Branson, N., Curean, V., Huebner, M., Dee, W., Connell, W., Chung, S. P., Hawkins-Hooker, A., Ektefaie, Y., et al. (2024) · doi:10.1101/2024.10.02.616248 · Cited by 37
Evaluating the learnability of single-cell large language models on multiple tasks
Yan, Y., Wang, X., & Song, D. (2026) · doi:10.21203/rs.3.rs-8919408/v1 · Cited by 1
Metadata
- Authors
- Wen, H., Tang, W., Dai, X., Ding, J., Jin, W., Xie, Y., & Tang, J.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2023-10-05
- OpenAlex
- W4387358535
- Cited by
- 71
- References
- 29
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: CellPLM: Pre-training of Cell Language Model Beyond Single Cells. https://app.cochat.ai/papers/10.1101/2023.10.03.560734
- The paper
- Wen, H., Tang, W., Dai, X., Ding, J., Jin, W., Xie, Y., & Tang, J. (2023). CellPLM: Pre-training of Cell Language Model Beyond Single Cells. bioRxiv (Cold Spring Harbor Laboratory). https://doi.org/10.1101/2023.10.03.560734
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.