The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index; 25 of the 27 sources its claims cite were checked for support.
Zero-shot evaluation reveals limitations of single-cell foundation models
Open accessAbstract
Foundation models such as scGPT and Geneformer have not been rigorously evaluated in a setting where they are used without any further training (i.e., zero-shot). Understanding the performance of models in zero-shot settings is critical to applications that exclude the ability to fine-tune, such as discovery settings where labels are unknown. Our evaluation of the zero-shot performance of Geneformer and scGPT suggests that, in some cases, these models may face reliability challenges and could be outperformed by simpler methods. Our findings underscore the importance of zero-shot evaluations in development and deployment of foundation models in single-cell research.
Overview
CoChat Summary
TL;DR
Zero-shot evaluation of Geneformer and scGPT on five single-cell datasets found they often underperform simpler baselines such as HVG, scVI, and Harmony for cell-type clustering and batch integration. The authors argue zero-shot testing is critical because fine-tuning can mask vulnerabilities.
Key findings
- In cell-type clustering, both Geneformer and scGPT performed worse than HVG, Harmony, and scVI on average BIO score; HVG outperformed Geneformer and scGPT across all metrics. scGPT performed better on the PBMC (12k) dataset compared to scVI, Harmony, and HVG, but was worse than both scVI and Harmony on AvgBIO score on other datasets.
- For batch integration, Geneformer underperformed relative to scGPT, Harmony, scVI, and HVG across most datasets and consistently ranked last across all metrics. The best batch integration scores for all datasets were achieved by selecting HVG; scVI and Harmony outperformed scGPT on Pancreas and PBMC, while scGPT outperformed them on Tabula Sapiens and Immune, both of which were used in scGPT pretraining.
- On the pretraining reconstruction task, scGPT without embeddings underperformed a naive mean baseline and only marginally improved with cell embeddings. Geneformer’s gene ranking showed modest correlations with true rankings, with median and best correlations of 0.56 and 0.95, respectively, across datasets.
- Pretraining improved scGPT over a random baseline on PBMC (12k), and the median score across datasets for the three scGPT models was greater than the random baseline; however, scGPT human slightly underperformed scGPT blood even for tissues beyond blood and bone marrow.
Methods
- The authors performed zero-shot evaluation of Geneformer and scGPT against HVG (2000 genes), scVI, and Harmony on five human tissue datasets: pancreas, two PBMC datasets, a cross-tissue immune cell atlas, and a multi-organ human cell atlas. scVI and Harmony were trained on each target dataset, while Geneformer and scGPT were used zero-shot.
- scGPT variants tested were pretrained on 814,000 kidney cells, 10.3 million blood and bone marrow cells, and 33 million non-cancerous human cells. Geneformer was pretrained on 27.4M human single-cell transcriptomes.
- Cell embedding quality was assessed with AvgBIO (mean of ASW, NMI, ARI), ASW, batch integration score, PCR score, and Average Batch Score. Pretraining reconstruction was assessed with MSE for scGPT and Pearson correlation between true and predicted rankings for Geneformer, using all genes unmasked as input.
Limitations
- The paper states that its approach was focused: it concentrated on specific models and a limited array of datasets that do not exhaustively represent all single-cell applications, and its datasets only include transcriptomic data.
- The paper states that some evaluation datasets were used in pretraining, confounding the ability to say whether performance trends are general or due to prior exposure; this specifically affects interpretation of scGPT improvements on Immune and Tabula Sapiens. It also notes that smaller scGPT models were trained on tissue-specific data, confounding whether differences are due to size or dataset composition.
- Evident from the text: scVI and Harmony were trained on target data while Geneformer and scGPT were zero-shot, so the comparison is not training-matched; the authors reason this reflects practical settings but it remains a difference in evaluation setup.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
19 Valid · 0 Not valid · 2 Not checked · 74 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our analysis indicates that pretraining provides a clear improvement in cell-type clustering on the PBMC (12k) dataset, and that the median score, calculated across datasets, for the three scGPT models is greater than that of the random baseline (Additional file 2: Fig. S5, Additional file 1: Table S1).”
§ Zero-shot evaluation reveals limitations of single-cell foundation models > Results and discussion
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observe that scGPT human and blood improve over random models for at least some datasets where scGPT kidney fails to, including the Immune and Tabula Sapiens datasets, suggesting that performance may improve with larger pretraining datasets.”
§ Zero-shot evaluation reveals limitations of single-cell foundation models > Results and discussion
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, HVG outperforms Geneformer and scGPT across all metrics (Additional file 2: Fig. S4, Additional file 1: Tables S1 and S2).”
§ Zero-shot evaluation reveals limitations of single-cell foundation models > Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“scGPT is comparable to scVI on the Tabula Sapiens, Pancreas, and PBMC (12k) datasets and outperforms Harmony on the Tabula Sapiens datasets with respect to ASW score (Additional file 2: Fig. S3, Additional file 1: Table S1).”
§ Zero-shot evaluation reveals limitations of single-cell foundation models > Results and discussion
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our analysis indicates that pretraining provides a clear improvement in cell-type clustering on the PBMC (12k) dataset, and that the median score, calculated across datasets, for the three scGPT models is greater than that of the random baseline (Additional file 2: Fig. S5, Additional file 1: Table S1).”
§ Zero-shot evaluation reveals limitations of single-cell foundation models > Results and discussion
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
52 more claims
Create a free account to see all 95 claims and to verify the 2 references still open.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 27
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Foundation models for single-cell and multi-omic biology, such as scGPT [20], Nicheformer [21], and EpiAgent [22, 23], produce high-quality continuous representations of biological entities that support cross-tissue transfer, perturbation inference, and regulatory feature learning.”
From The next paradigm in bioinformatics: a review of multi-agent systems and foundational models for end-to-end scientific discovery (2026) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical benchmarking studies have documented substantial performance degradation for biological foundation models under cross-platform transfer, preprocessing variability, and rare-population evaluation [20–24].”
From The next paradigm in bioinformatics: a review of multi-agent systems and foundational models for end-to-end scientific discovery (2026) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have provided initial benchmarks comparing foundation models, other deep learning architectures, and simple baselines for perturbation effect prediction and reported that complex models do not consistently outperform the simpler alternatives (22–26).”
From A systematic comparison of single-cell perturbation response prediction models (2026) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We first examine recent foundational models that produce transferable representations across omic modalities, such as scGPT, Nicheformer, and EpiAgent, and discuss their architectural choices, training regimes, and interpretability constraints.”
From The next paradigm in bioinformatics: a review of multi-agent systems and foundational models for end-to-end scientific discovery (2026) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical benchmarking studies have shown that performance improvements observed in standard evaluation settings can decrease substantially when models are tested across different laboratories, sequencing platforms, or preprocessing pipelines [20–23].”
From The next paradigm in bioinformatics: a review of multi-agent systems and foundational models for end-to-end scientific discovery (2026) · cites this paper as [22]
Experimental — accuracy not yet measured
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Zero-shot evaluation reveals limitations of single-cell foundation models > Results and discussion
“Harmony ranking last for batch integration but second for principal component regression (PCR) score”
§ Zero-shot evaluation reveals limitations of single-cell foundation models > Results and discussion
“Geneformer consistently ranks at the bottom across all metrics”
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Fast, sensitive and accurate integration of single-cell data with Harmony
Korsunsky, I., Millard, N., Fan, J., Slowikowski, K., Zhang, F., Wei, K., Baglaenko, Y., Brenner, M., Loh, P.-R., & Raychaudhuri, S. (2019) · Nature Methods · doi:10.1038/s41592-019-0619-0 · Cited by 10,488 · cited 2×
“Both scGPT and Geneformer produce cell embeddings intended to project potentially noisy gene expression measurements to a more biologically relevant latent space [18– 21], and then these cell embeddings are fine-tuned for cell type classification.”
Massively parallel digital transcriptional profiling of single cells
Zheng, G. X. Y., Terry, J. M., Belgrader, P., Ryvkin, P., Bent, Z. W., Wilson, R., Ziraldo, S. B., Wheeler, T. D., McDermott, G. P., Zhu, J., et al. (2017) · Nature Communications · doi:10.1038/ncomms14049 · Cited by 7,353 · cited 1×
“These datasets include samples from the pancreas [25], two sets of peripheral blood mononuclear cells (PBMCs) [30, 31], a cross-tissue immune cell atlas [32], and a multi-organ human cell atlas [33].”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 1×
“To tackle this, we propose the creation of benchmark tasks and datasets reserved exclusively for model evaluation that should never be used to pretrain any future model, as has been done for large language models (LLMs) [26–29].”
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Deep generative modeling for single-cell transcriptomics
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., & Yosef, N. (2018) · Nature Methods · doi:10.1038/s41592-018-0229-2 · Cited by 2,530 · cited 1×
“In addition, we compared all methods to scVI, a scalable generative model [20], and Harmony, a method for adjusting shared embedding space [21], which we trained on each individual dataset.”
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417
HellaSwag: Can a Machine Really Finish Your Sentence?Preprint
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07830 · Cited by 2,074
Cited by
Ahlmann-Eltze, C., Huber, W., & Anders, S. (2025) · Nature Methods · doi:10.1038/s41592-025-02772-6 · Cited by 238
“Our results are in line with previously published benchmarks that assessed the performance of foundation models for other tasks and found negligible benefits compared to simpler approaches 17–19 .”
Nicheformer: a foundation model for single-cell and spatial omics
Tejada-Lapuerta, A., Schaar, A. C., Gutgesell, R., Palla, G., Halle, L., Minaeva, M., Vornholz, L., Dony, L., Drummer, F., Richter, T., et al. (2025) · Nature Methods · doi:10.1038/s41592-025-02814-z · Cited by 100
Single-cell foundation models: bringing artificial intelligence into cell biology
Baek, S., Song, K., & Lee, I. (2025) · Experimental & Molecular Medicine · doi:10.1038/s12276-025-01547-5 · Cited by 46
“Importantly, fi ne-tuning scFMs on task-speci fi c datasets generally leads to substantial gains in performance and enables more complex applications that may not be feasible in zero-shot settings 2,41 .”
Multimodal learning enables chat-based exploration of single-cell data
Schaefer, M., Peneder, P., Malzl, D., Lombardo, S. D., Peycheva, M., Burton, J., Hakobyan, A., Sharma, V., Krausgruber, T., Sin, C., et al. (2025) · Nature Biotechnology · doi:10.1038/s41587-025-02857-9 · Cited by 81
Embeddings from language models are good learners for single-cell data analysis
Liu, T., Chen, T., Zheng, W., Luo, X., Chen, Y. T., & Zhao, H. (2026) · Patterns · doi:10.1016/j.patter.2025.101431 · Cited by 44
“Recent benchmarking studies 16,17 have highlighted limitations in the scalability, robustness, and generalizability of current single-cell FMs.”
Metadata
- Authors
- Kedzierska, K. Z., Crawford, L., Amini, A. P., & Lu, A. X.
- Affiliations
- University of Oxford (GB)
- Microsoft Research New England (United States) (US)
- Microsoft (United States) (US)
- Corresponding author
- Lu, Alex X.
- Venue
- Genome Biology
- Published
- 2025-04-18
- Volume
- 26
- Issue
- 1
- Pages
- 101-101
- ISSN
- 1465-6906 (ISSN-L)
- 1474-760X
- PMC
- PMC12007350
- OpenAlex
- W4409566559
- Cited by
- 109
- References
- 44
- Citations by year
- 2026: 78 · 2025: 30
- FWCI
- 32.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Genome biology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Microsoft
- Wellcome Trust (203141/Z/16/Z)
- NIHR Oxford Biomedical Research Centre
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Keywords
- zero-shot evaluation, single-cell foundation models, scGPT, Geneformer, single-cell RNA sequencing, model reliability, single-cell transcriptomics
- MeSH terms
- Humans, Software, Single-Cell Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Zero-shot evaluation reveals limitations of single-cell foundation models. https://app.cochat.ai/papers/10.1186/s13059-025-03574-x
- The paper
- Kedzierska, K. Z., Crawford, L., Amini, A. P., & Lu, A. X. (2025). Zero-shot evaluation reveals limitations of single-cell foundation models. Genome Biology. https://doi.org/10.1186/s13059-025-03574-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.