Evaluating the learnability of single-cell large language models on multiple tasks
Open accessPublished version of a preprint
Abstract
The rise of single-cell foundation models (scFMs) has sparked interest in their potential to unify diverse biological tasks. However, their practical utility and the validity of scaling laws-the assumption that performance improves with model and data size-remain under-examined. Here, we systematically evaluate two representative scFMs, Geneformer and scGPT, across perturbation prediction and cell type annotation tasks. Our findings suggest that the benefits of large-scale pretraining are strongly task-dependent, conferring substantial advantages in cell type annotation but limited gains in perturbation prediction. Furthermore, our results indicate that increasing model size does not guarantee improved performance and can even be detrimental, challenging the "bigger is better" paradigm for the models and tasks examined here. By comparing model performance on real versus synthetic data with different levels of complexity, our analysis suggests that for perturbation prediction, the tested scFMs may capture little more than simple summary statistics, suggesting limited capacity to learn complex biological interactions within our experimental design. Based on our evaluation of Geneformer and scGPT, these results highlight the need to move beyond scaling and toward developing models that integrate deeper biological knowledge. We suggest that a renewed focus on task-specific architectures and biologically-informed priors may be critical for unlocking the true potential of foundation models in single-cell biology.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Aran, D., Looney, A. P., Liu, L., Wu, E., Fong, V., Hsu, A., Chak, S., Naikawadi, R. P., Wolters, P. J., Abate, A. R., et al. (2019) · Nature Immunology · doi:10.1038/s41590-018-0276-y · Cited by 4,984 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“This paradigm, inspired by the success of foundation models in natural language processing and computer vision, holds the promise of unlocking scaling laws, where model performance systematically improves with increasing data and parameter counts [11–13].”
A pan-cancer single-cell transcriptional atlas of tumor infiltrating myeloid cells
Cheng, S., Li, Z., Gao, R., Xing, B., Gao, Y., Yang, Y., Qin, S., Zhang, L., Ouyang, H., Du, P., et al. (2021) · Cell · doi:10.1016/j.cell.2021.01.010 · Cited by 1,601 · see citing works
Self-citationscGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., & Wang, B. (2023) · doi:10.1101/2023.04.30.538439 · Cited by 1,383 · see citing works
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · see citing works
Adamson, B., Norman, T. M., Jost, M., Cho, M. Y., Nuñez, J. K., Chen, Y., Villalta, J. E., Gilbert, L. A., Horlbeck, M. A., Hein, M. Y., et al. (2016) · Cell · doi:10.1016/j.cell.2016.11.048 · Cited by 1,193 · see citing works
Transfer learning enables predictions in network biology
Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., Al Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng, Z., Liu, X. S., et al. (2023) · Nature · doi:10.1038/s41586-023-06139-9 · Cited by 1,193 · see citing works
Cross-tissue immune cell analysis reveals tissue-specific features in humans
Conde, C. D., Xu, C., Jarvis, L. B., Rainbow, D. B., Wells, S. B., Gomes, T., Howlett, S. K., Suchánek, O., Polański, K., King, H. W., et al. (2022) · Science · doi:10.1126/science.abl5197 · Cited by 1,165 · see citing works
The Tabula Sapiens: A multiple-organ, single-cell transcriptomic atlas of humans
The Tabula Sapiens Consortium*, Jones, R. C., Karkanias, J., Krasnow, M. A., Pisco, A. O., Quake, S. R., Salzman, J., Yosef, N., Bulthaup, B., Brown, P., et al. (2022) · Science · doi:10.1126/science.abl4896 · Cited by 1,056 · see citing works
Best practices for single-cell analysis across modalities
Heumos, L., Schaar, A. C., Lance, C., Litinetskaya, A., Drost, F., Zappia, L., Lücken, M. D., Strobl, D. C., Henao, J., Curion, F., et al. (2023) · Nature Reviews Genetics · doi:10.1038/s41576-023-00586-w · Cited by 1,051 · see citing works
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · see citing works
scGen predicts single-cell perturbation responses
Lotfollahi, M., Wolf, F. A., & Theis, F. J. (2019) · Nature Methods · doi:10.1038/s41592-019-0494-8 · Cited by 661 · see citing works
Large Scale Foundation Model on Single-cell Transcriptomics
Hao, M., Gong, J., Zeng, X., Liu, C., Guo, Y., Cheng, X., Wang, T., Ma, J., Song, L., & Zhang, X. (2023) · doi:10.1101/2023.05.29.542705 · Cited by 639 · see citing works
Neuronal vulnerability and multilineage diversity in multiple sclerosis
Schirmer, L., Velmeshev, D., Holmqvist, S., Kaufmann, M., Werneburg, S., Jung, D., Vistnes, S., Stockley, J. H., Young, A., Steindel, M., et al. (2019) · Nature · doi:10.1038/s41586-019-1404-z · Cited by 638 · see citing works
Large-scale foundation model on single-cell transcriptomics
Hao, M., Gong, J., Zeng, X., Liu, C., Guo, Y., Cheng, X., Wang, T., Ma, J., Zhang, X., & Song, L. (2024) · Nature Methods · doi:10.1038/s41592-024-02305-7 · Cited by 628 · cited 1× · see citing works
“Recent years have witnessed an explosion of scFMs, including scGPT [9], Geneformer [10], scFounda-tion [14], scBERT [15], UCE [16], CellPLM [17], GenePT [18], and GeneCompass [19], among others.”
Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., & Yao, J. (2021) · doi:10.1101/2021.12.05.471261 · Cited by 617 · see citing works
Single-cell epigenomics: Recording the past and predicting the future
Kelsey, G., Stegle, O., & Reik, W. (2017) · Science · doi:10.1126/science.aan6826 · Cited by 415 · see citing works
Exploring genetic interaction manifolds constructed from rich single-cell phenotypes
Norman, T. M., Horlbeck, M. A., Replogle, J. M., Ge, A. Y., Xu, A., Jost, M., Gilbert, L. A., & Weissman, J. S. (2019) · Science · doi:10.1126/science.aax4438 · Cited by 392 · see citing works
Predicting transcriptional outcomes of novel multigene perturbations with GEARS
Roohani, Y., Huang, K., & Leskovec, J. (2024) · Nature Biotechnology · doi:10.1038/s41587-023-01905-6 · Cited by 355 · see citing works
Predicting cellular responses to complex perturbations in high‐throughput screens
Lotfollahi, M., Klimovskaia Susmelj, A., De Donno, C., Hetzel, L., Ji, Y., Ibarra, I. L., Srivatsan, S. R., Naghipourfar, M., Daza, R. M., Martin, B., et al. (2023) · Molecular Systems Biology · doi:10.15252/msb.202211517 · Cited by 307 · see citing works
Ahlmann-Eltze, C., Huber, W., & Anders, S. (2025) · Nature Methods · doi:10.1038/s41592-025-02772-6 · Cited by 238 · cited 3× · see citing works
“Despite these critical findings, the reasons why large-scale models often fail to outperform smaller, specialized counterparts remain poorly understood [31–34].”
Transformer for one stop interpretable cell type annotation
Chen, J., Xu, H., Tao, W., Chen, Z., Zhao, Y., & Han, J.-D. J. (2023) · Nature Communications · doi:10.1038/s41467-023-35923-4 · Cited by 189 · see citing works
Simple and effective embedding model for single-cell biology built from ChatGPT
Chen, Y. T., & Zou, J. (2024) · Nature Biomedical Engineering · doi:10.1038/s41551-024-01284-6 · Cited by 172 · see citing works
The Human Cell Atlas from a cell census to a unified foundation model
Rood, J., Wynne, S., Robson, L., Hupalowska, A., Randell, J., Teichmann, S. A., & Regev, A. (2024) · Nature · doi:10.1038/s41586-024-08338-4 · Cited by 168 · see citing works
Universal Cell Embeddings: A Foundation Model for Cell BiologyPreprint
Rosen, Y., Roohani, Y., Agrawal, A., Samotorcan, L., Quake, S. R., & Leskovec, J. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.11.28.568918 · Cited by 165 · see citing works
Yang, X., Liu, G., Feng, G., Bu, D., Wang, P., Jiang, J., Chen, S., Yang, Q., Zhang, Y., Man, Z., et al. (2023) · doi:10.1101/2023.09.26.559542 · Cited by 148 · see citing works
Self-citationscDesign3 generates realistic in silico data for multimodal single-cell and spatial omics
Song, D., Wang, Q., Yan, G., Liu, T., Sun, T., & Li, J. J. (2023) · Nature Biotechnology · doi:10.1038/s41587-023-01772-1 · Cited by 148 · see citing works
Self-citationYang, X., Liu, G., Feng, G., Bu, D., Wang, P., Jiang, J., Chen, S., Yang, Q., Miao, H., Zhang, Y., et al. (2024) · Cell Research · doi:10.1038/s41422-024-01034-y · Cited by 148 · see citing works
Self-citationZero-shot evaluation reveals limitations of single-cell foundation models
Kedzierska, K. Z., Crawford, L., Amini, A. P., & Lu, A. X. (2025) · Genome Biology · doi:10.1186/s13059-025-03574-x · Cited by 109 · see citing works
CellPLM: Pre-training of Cell Language Model Beyond Single CellsPreprint
Wen, H., Tang, W., Dai, X., Ding, J., Jin, W., Xie, Y., & Tang, J. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.03.560734 · Cited by 85 · see citing works
Evaluating the Utilities of Foundation Models in Single‐Cell Data Analysis
Liu, T., Li, K., Wang, Y., Li, H., & Zhao, H. (2026) · Advanced Science · doi:10.1002/advs.202514490 · Cited by 76 · see citing works
Evaluating the Utilities of Foundation Models in Single-cell Data Analysis
Liu, T., Li, K., Wang, Y., Li, H., & Zhao, H. (2023) · doi:10.1101/2023.09.08.555192 · Cited by 62 · see citing works
Assessing the limits of zero-shot foundation models in single-cell biology
Kedzierska, K. Z., Crawford, L., Amini, A. P., & Lu, A. X. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.16.561085 · Cited by 61 · see citing works
Viñas Torné, R., Wiatrak, M., Piran, Z., Fan, S., Jiang, L., Teichmann, S. A., Nitzan, M., & Brbić, M. (2026) · Nature Biotechnology · doi:10.1038/s41587-025-02777-8 · Cited by 58 · see citing works
Benchmarking foundation cell models for post-perturbation RNA-seq prediction
Csendes, G., Sanz, G., Szalay, K. Z., & Szalai, B. (2025) · BMC Genomics · doi:10.1186/s12864-025-11600-2 · Cited by 47 · cited 1× · see citing works
“…have found scFMs to be inconsistent across datasets, with some models underperforming simpler baselines in certain settings [26, 27], and benchmarks of perturbation prediction have similarly reported that foundation models can be outperformed by straight-forward baseline approaches [28].”
Benchmarking algorithms for generalizable single-cell perturbation response prediction
Wei, Z., Wang, Y., Gao, Y., Wang, S., Li, P., Si, D., Gao, Y., Wu, S., Li, D., Dong, K., et al. (2026) · Nature Methods · doi:10.1038/s41592-025-02980-0 · Cited by 42 · see citing works
PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction
Wenteler, A., Occhetta, M., Branson, N., Curean, V., Huebner, M., Dee, W., Connell, W., Chung, S. P., Hawkins-Hooker, A., Ektefaie, Y., et al. (2024) · doi:10.1101/2024.10.02.616248 · Cited by 37 · see citing works
Biology-driven insights into the power of single-cell foundation models
Wu, J., Ye, Q., Wang, Y., Hu, R., Zhu, Y., Yin, M., Wang, T., Wang, J., Hsieh, C., & Hou, T. (2025) · Genome biology · doi:10.1186/s13059-025-03781-6 · Cited by 30 · see citing works
DenAdel, A., Hughes, M., Thoutam, A., Gupta, A., Navia, A. W., Fusi, N., Raghavan, S., Winter, P. S., Amini, A. P., & Crawford, L. A. (2026) · Nature Methods · doi:10.1038/s41592-026-03120-y · Cited by 28 · see citing works
Deeper evaluation of a single-cell foundation model
Boiarsky, R., Singh, N. M., Buendia, A., Amini, A. P., Getz, G., & Sontag, D. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00949-w · Cited by 25 · see citing works
BioLLM: A standardized framework for integrating and benchmarking single-cell foundation models
Qiu, P. S., Chen, Q., Qin, H., Fang, S., Zhang, Y., Zhang, Y., Xia, T., Cao, L., Zhang, Y., Fang, X., et al. (2025) · Patterns · doi:10.1016/j.patter.2025.101326 · Cited by 21 · see citing works
scDrugMap: benchmarking large foundation models for drug response prediction
Wang, Q., Pan, Y., Zhou, M., Tang, Z., Wang, Y., Wang, G., & Song, Q. (2025) · Nature Communications · doi:10.1038/s41467-025-67481-2 · Cited by 12 · see citing works
A Systematic Evaluation of Single-Cell Foundation Models on Cell-Type Classification Task
Steiner, N., Li, Z., Vosoughi, O., Schrader, J., Roy, S., Nejdl, W., & Tang, M. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3708811 · Cited by 6 · see citing works
Overcoming barriers to the wide adoption of single-cell large language models in biomedical research
Xie, F., Zhao, B., Xu, S., Wang, Z., Moon, J. J., & Garmire, L. X. (2025) · Nature Biotechnology · doi:10.1038/s41587-025-02846-y · Cited by 4 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Cited by
Sorted by citations
Zhao, X., Pang, Y. Y., Fu, T. Y., Yang, C. N., & Moi, S. H. (2026) · IEEE Access · doi:10.1109/access.2026.3744403
- Authors
- Yan, Y., Wang, X., & Song, D.
- Affiliations
- University of California, Los Angeles (US)
- Rutgers, The State University of New Jersey (US)
- UConn Health (US)
- Corresponding author
- Yan, Yu
- Song, Dongyuan
- Venue
- BMC Genomics
- Published
- 2026-06-05
- Volume
- 27
- Issue
- 1
- ISSN
- 1471-2164 (ISSN-L)
- PMC
- PMC13465196
- OpenAlex
- W7163700209
- Cited by
- 1
- References
- 43
- Citations by year
- 2026: 1
- FWCI
- 2.07
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Genomics (publishedVersion, cc-by-nc-nd)
- Research Square (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Artificial Intelligence Research Resource (NAIRR) Pilot
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Keywords
- single-cell foundation models, large language models, Geneformer, scGPT, scaling laws, cell type annotation, perturbation prediction, model size, single-cell biology
- MeSH terms
- Large Language Models, Animals, Humans, Single-Cell Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
