Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 47Single-cell and spatial transcriptomicsGene expression and cancer classificationGene Regulatory Network Analysis

The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index; 9 of the 10 sources its claims cite were checked for support.

Benchmarking foundation cell models for post-perturbation RNA-seq prediction

Open access

Abstract

Accurately predicting cellular responses to perturbations is essential for understanding cell behaviour in both healthy and diseased states. While perturbation data is ideal for building such predictive models, its availability is considerably lower than baseline (non-perturbed) cellular data. To address this limitation, several foundation cell models have been developed using large-scale single-cell gene expression data. These models are fine-tuned after pre-training for specific tasks, such as predicting post-perturbation gene expression profiles, and are considered state-of-the-art for these problems. However, proper benchmarking of these models remains an unsolved challenge. In this study, we benchmarked two recently published foundation models, scGPT and scFoundation, against baseline models. Surprisingly, we found that even the simplest baseline model—taking the mean of training examples—outperformed scGPT and scFoundation. Furthermore, basic machine learning models that incorporate biologically meaningful features outperformed scGPT by a large margin. Additionally, we identified that the current Perturb-Seq benchmark datasets exhibit low perturbation-specific variance, making them suboptimal for evaluating such models. Our results highlight important limitations in current benchmarking approaches and provide insights into more effectively evaluating post-perturbation gene expression prediction models.

Overview

CoChat Summary

TL;DR

In a benchmark of scGPT and scFoundation on four Perturb-seq datasets, the trivial Train Mean baseline and a Random Forest using Gene Ontology features outperformed both foundation models on differential-expression correlation metrics. The authors attribute this to low perturbation-specific variance in current benchmark datasets and argue that existing benchmarks are suboptimal for evaluating post-perturbation RNA-seq prediction.

Key findings

  • In differential expression space (Pearson Delta), Train Mean outperformed scGPT and scFoundation across Adamson, Norman, Replogle K562, and Replogle RPE1: Train Mean 0.711, 0.557, 0.373, 0.628; scGPT 0.641, 0.554, 0.327, 0.596; scFoundation 0.552, 0.459, 0.269, 0.471.
  • Random Forest with GO features outperformed foundation models by a large margin: 0.739, 0.586, 0.480, 0.648 across the four datasets, respectively.
  • Random Forest using pretrained foundation embeddings performed better than the fine-tuned foundation models themselves, especially for scGPT (0.727, 0.583, 0.421, 0.635), but still weaker than RF with GO features; RF with scELMO features was similar to GO-based RF (0.706, 0.663, 0.471, 0.651).
  • The benchmark datasets contain many single cells (~70,000 in Adamson to ~160,000 in Replogle datasets) but few distinct perturbations (87 Adamson, 284 Norman, 1,093 Replogle K562, 1,544 Replogle RPE1); Adamson had high median pairwise differential-expression correlation (0.662), while Replogle K562 had lower median correlation (0.117).

Methods

  • Benchmark datasets (Adamson, Norman, two Replogle subsets) were downloaded and processed with GEARS’ cell-gears v0.0.1; expression values were normalized to 10,000 reads and log transformed, top 20 DE genes were taken from the original GEARS publication, and splits followed the GEARS Perturbation Exclusive setup.
  • scGPT was reproduced using a fork of scGPT v0.2.1 (commit 7301b51) with default hyperparameters and fine-tuned for 15 epochs; scFoundation used repository commit 69b0710 and was fine-tuned for 10 epochs with effective batch size 32; both trained on Nvidia A100 GPUs.
  • Baselines included Train Mean, Random Forest Regressor, Elastic Net, and k-Nearest-Neighbors Regressor, using prior-knowledge features (GO PCA to 256 components, scGPT/scFoundation/scELMO embeddings) and pseudo-bulked expression targets; evaluation used Pearson, Pearson Delta, Pearson Delta DE, and Pearson Delta without target genes.

Limitations

  • Stated by the paper: current Perturb-seq benchmark datasets exhibit low perturbation-specific variance, making them suboptimal for evaluating models; Adamson and Replogle RPE1 show high similarity between perturbations, and current benchmarks focus exclusively on the PEX problem while neglecting the CEX problem.
  • Stated by the paper: single-cell data did not provide an obvious advantage over pseudo-bulked data in this benchmark, and single-cell profiles are subject to technical noise such as dropout events; only scGPT and scFoundation were tested, though other foundation models may face similar limitations.
  • Evident from the text: evaluation is limited to four Perturb-seq datasets and PEX splits, and the top-20 DE metric can be influenced by inclusion of the CRISPR target gene, since removing target genes decreased scGPT’s performance in that space.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s12864-025-11600-2

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Benchmarking foundation and baseline models (A) Schematic representation of benchmark pipeline (B) Evaluation on the Adamson dataset: Pearson delta metrics (y axis) for scGPT, scFoundation, Train Mean and Random Forest Regression model with different features (x axis). Main groups of models a colour coded. C Evaluation on the Norman dataset (D) Evaluation on the Replogle K562 dataset (E) Evaluation on the Replogle RPE1 dataset
Fig. 1 Benchmarking foundation and baseline models (A) Schematic representation of benchmark pipeline (B) Evaluation on the Adamson dataset: Pearson delta metrics (y axis) for scGPT, scFoundation, Train Mean and Random Forest Regression model with different features (x axis). Main groups of models a colour coded. C Evaluation on the Norman dataset (D) Evaluation on the Replogle K562 dataset (E) Evaluation on the Replogle RPE1 datasetCC BY · PubMed Central
Fig. 2 Composition of standard benchmark datasets (A) Number of single cells (control/perturbed, colour code) in the four benchmark datasets. y axis is log10 scaled (B) Number of distinct perturbations in the four benchmark datasets (C) Correlation heatmaps for pseudo-bulk differential expression signatures for Adamson (left), and Replogle K562 (right) datasets. The black lines indicate the separation between training and test sets. Some samples (perturbations) are labelled on the x and y axes. D Distribution of pairwise Pearson correlations (y axis) of differential expression profiles for the benchmark datasets (x axis). E Comparison between median intra-dataset correlations (x axis) and Pearson Delta metrics difference between best and Train Mean models (y axis), colour coded by the benchmark dataset
Fig. 2 Composition of standard benchmark datasets (A) Number of single cells (control/perturbed, colour code) in the four benchmark datasets. y axis is log10 scaled (B) Number of distinct perturbations in the four benchmark datasets (C) Correlation heatmaps for pseudo-bulk differential expression signatures for Adamson (left), and Replogle K562 (right) datasets. The black lines indicate the separation between training and test sets. Some samples (perturbations) are labelled on the x and y axes. D Distribution of pairwise Pearson correlations (y axis) of differential expression profiles for the benchmark datasets (x axis). E Comparison between median intra-dataset correlations (x axis) and Pearson Delta metrics difference between best and Train Mean models (y axis), colour coded by the benchmark datasetCC BY · PubMed Central

Verification

Claims the paper makes

6 Valid · 0 Not valid · 1 Not checked · 42 Own results and interpretations

20 more claims

Create a free account to see all 49 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 8
  • It may state two separate claims as one. 17
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 35 references

Cited by

See all 47

Metadata

Authors
Csendes, G., Sanz, G., Szalay, K. Z., & Szalai, B.
Corresponding author
Szalai, Bence
Venue
BMC Genomics
Published
2025-04-23
Volume
26
Issue
1
Pages
393-393
ISSN
1471-2164 (ISSN-L)
OpenAlex
W4409705874
Cited by
47
References
35
Citations by year
2026: 29 · 2025: 18
FWCI
14.03
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Genomics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
Keywords
benchmarking, scGPT, single-cell RNA sequencing, Perturb-seq, single-cell gene expression, model evaluation, baseline model
MeSH terms
Machine Learning, RNA-Seq, Humans, Models, Biological, Sequence Analysis, RNA, Computational Biology, Benchmarking, Gene Expression Profiling, Single-Cell Analysis, Transcriptome

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Benchmarking foundation cell models for post-perturbation RNA-seq prediction. https://app.cochat.ai/papers/10.1186/s12864-025-11600-2
The paper
Csendes, G., Sanz, G., Szalay, K. Z., & Szalai, B. (2025). Benchmarking foundation cell models for post-perturbation RNA-seq prediction. BMC Genomics. https://doi.org/10.1186/s12864-025-11600-2

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.