Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index; 25 of the 27 sources its claims cite were checked for support.

Zero-shot evaluation reveals limitations of single-cell foundation models

Open access
Kedzierska, Kasia Z., Crawford, Lorin, Amini, Ava P., Lu, Alex X. · 2025 · Genome Biology

Abstract

Foundation models such as scGPT and Geneformer have not been rigorously evaluated in a setting where they are used without any further training (i.e., zero-shot). Understanding the performance of models in zero-shot settings is critical to applications that exclude the ability to fine-tune, such as discovery settings where labels are unknown. Our evaluation of the zero-shot performance of Geneformer and scGPT suggests that, in some cases, these models may face reliability challenges and could be outperformed by simpler methods. Our findings underscore the importance of zero-shot evaluations in development and deployment of foundation models in single-cell research.

Overview

CoChat Summary

TL;DR

Zero-shot evaluation of Geneformer and scGPT on five single-cell datasets found they often underperform simpler baselines such as HVG, scVI, and Harmony for cell-type clustering and batch integration. The authors argue zero-shot testing is critical because fine-tuning can mask vulnerabilities.

Key findings

  • In cell-type clustering, both Geneformer and scGPT performed worse than HVG, Harmony, and scVI on average BIO score; HVG outperformed Geneformer and scGPT across all metrics. scGPT performed better on the PBMC (12k) dataset compared to scVI, Harmony, and HVG, but was worse than both scVI and Harmony on AvgBIO score on other datasets.
  • For batch integration, Geneformer underperformed relative to scGPT, Harmony, scVI, and HVG across most datasets and consistently ranked last across all metrics. The best batch integration scores for all datasets were achieved by selecting HVG; scVI and Harmony outperformed scGPT on Pancreas and PBMC, while scGPT outperformed them on Tabula Sapiens and Immune, both of which were used in scGPT pretraining.
  • On the pretraining reconstruction task, scGPT without embeddings underperformed a naive mean baseline and only marginally improved with cell embeddings. Geneformer’s gene ranking showed modest correlations with true rankings, with median and best correlations of 0.56 and 0.95, respectively, across datasets.
  • Pretraining improved scGPT over a random baseline on PBMC (12k), and the median score across datasets for the three scGPT models was greater than the random baseline; however, scGPT human slightly underperformed scGPT blood even for tissues beyond blood and bone marrow.

Methods

  • The authors performed zero-shot evaluation of Geneformer and scGPT against HVG (2000 genes), scVI, and Harmony on five human tissue datasets: pancreas, two PBMC datasets, a cross-tissue immune cell atlas, and a multi-organ human cell atlas. scVI and Harmony were trained on each target dataset, while Geneformer and scGPT were used zero-shot.
  • scGPT variants tested were pretrained on 814,000 kidney cells, 10.3 million blood and bone marrow cells, and 33 million non-cancerous human cells. Geneformer was pretrained on 27.4M human single-cell transcriptomes.
  • Cell embedding quality was assessed with AvgBIO (mean of ASW, NMI, ARI), ASW, batch integration score, PCR score, and Average Batch Score. Pretraining reconstruction was assessed with MSE for scGPT and Pearson correlation between true and predicted rankings for Geneformer, using all genes unmasked as input.

Limitations

  • The paper states that its approach was focused: it concentrated on specific models and a limited array of datasets that do not exhaustively represent all single-cell applications, and its datasets only include transcriptomic data.
  • The paper states that some evaluation datasets were used in pretraining, confounding the ability to say whether performance trends are general or due to prior exposure; this specifically affects interpretation of scGPT improvements on Immune and Tabula Sapiens. It also notes that smaller scGPT models were trained on tissue-specific data, confounding whether differences are due to size or dataset composition.
  • Evident from the text: scVI and Harmony were trained on target data while Geneformer and scGPT were zero-shot, so the comparison is not training-matched; the authors reason this reflects practical settings but it remains a difference in evaluation setup.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13059-025-03574-x

Cites

52 of 52 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Evaluation of the cell embedding space generated by the models. A Overview of the evaluation setup. We compare Geneformer and scGPT to scVI, Harmony, and the selection of highly variable genes (HVG) on five diverse datasets. B Average BIO score for HVG and embeddings from Harmony, scVI, scGPT, and Geneformer. C, D Visualization of the UMAP projections of the Pancreas (16k) dataset using the cell embedding space generated by the models. Cells are color-coded by cell type (C) and batch (D). E Average batch score for HVG and embeddings from Harmony, scVI, scGPT, and Geneformer. Dashed line in B and E signifies the median calculated across the datasets
Fig. 1 Evaluation of the cell embedding space generated by the models. A Overview of the evaluation setup. We compare Geneformer and scGPT to scVI, Harmony, and the selection of highly variable genes (HVG) on five diverse datasets. B Average BIO score for HVG and embeddings from Harmony, scVI, scGPT, and Geneformer. C, D Visualization of the UMAP projections of the Pancreas (16k) dataset using the cell embedding space generated by the models. Cells are color-coded by cell type (C) and batch (D). E Average batch score for HVG and embeddings from Harmony, scVI, scGPT, and Geneformer. Dashed line in B and E signifies the median calculated across the datasetsCC BY · PubMed Central
Fig. 2 Performance comparison of scGPT and Geneformer in gene expression reconstruction. A–C Reconstruction of the expression in the Immune (330k) dataset: A scGPT gene expression prediction (GEP) under the masked language modeling (MLM) objective. B scGPT gene expression prediction from cell embeddings (GEPC). C Geneformer MLM output of the predicted expression ranking (y-axis) versus the true input expression ranking (x-axis). D Mean squared error (MSE) comparison for scGPT objectives. Mean and standard deviation range are shown as points and solid lines, respectively, and a median MSE for mean-based reconstruction is shown as a dashed line. E Pearson’s correlation of input and predicted expression ranking for both Geneformer and for average gene rankings. Mean and standard deviation range are shown as points and solid lines, respectively, and a median correlation for average ranking is shown as a dashed line
Fig. 2 Performance comparison of scGPT and Geneformer in gene expression reconstruction. A–C Reconstruction of the expression in the Immune (330k) dataset: A scGPT gene expression prediction (GEP) under the masked language modeling (MLM) objective. B scGPT gene expression prediction from cell embeddings (GEPC). C Geneformer MLM output of the predicted expression ranking (y-axis) versus the true input expression ranking (x-axis). D Mean squared error (MSE) comparison for scGPT objectives. Mean and standard deviation range are shown as points and solid lines, respectively, and a median MSE for mean-based reconstruction is shown as a dashed line. E Pearson’s correlation of input and predicted expression ranking for both Geneformer and for average gene rankings. Mean and standard deviation range are shown as points and solid lines, respectively, and a median correlation for average ranking is shown as a dashed lineCC BY · PubMed Central

Verification

Claims the paper makes

19 Valid · 0 Not valid · 2 Not checked · 74 Own results and interpretations

52 more claims

Create a free account to see all 95 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 20
  • Probably not a substantial claim. 27

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

3 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 44 references

Cited by

See all 109

Metadata

Authors
Kedzierska, K. Z., Crawford, L., Amini, A. P., & Lu, A. X.
Affiliations
  • University of Oxford (GB)
  • Microsoft Research New England (United States) (US)
  • Microsoft (United States) (US)
Corresponding author
Lu, Alex X.
Venue
Genome Biology
Published
2025-04-18
Volume
26
Issue
1
Pages
101-101
ISSN
  • 1465-6906 (ISSN-L)
  • 1474-760X
OpenAlex
W4409566559
Cited by
109
References
44
Citations by year
2026: 78 · 2025: 30
FWCI
32.24
Type
journal article
Language
en
Open access
Licence
Locations
  • Genome biology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Microsoft
  • Wellcome Trust (203141/Z/16/Z)
  • NIHR Oxford Biomedical Research Centre
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
Keywords
zero-shot evaluation, single-cell foundation models, scGPT, Geneformer, single-cell RNA sequencing, model reliability, single-cell transcriptomics
MeSH terms
Humans, Software, Single-Cell Analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Zero-shot evaluation reveals limitations of single-cell foundation models. https://app.cochat.ai/papers/10.1186/s13059-025-03574-x
The paper
Kedzierska, K. Z., Crawford, L., Amini, A. P., & Lu, A. X. (2025). Zero-shot evaluation reveals limitations of single-cell foundation models. Genome Biology. https://doi.org/10.1186/s13059-025-03574-x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.