CoChat

A systematic comparison of single-cell perturbation response prediction models

Open access

The DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index; 27 of the 33 sources its claims cite were checked for support.

Abstract

Predicting single-cell transcriptional responses to perturbations is central to dissecting gene regulation and accelerating therapeutic design, yet the field lacks a rigorous, task-spanning assessment of model behavior. We present a large-scale benchmark of 13 representative methods and baselines across 25 datasets spanning diverse perturbation modalities and species, including two primary immune-cell drug-response resources. We evaluated three core tasks-generalization to unseen single-gene perturbations, prediction of combinatorial interactions, and transfer across cell types-using 24 metrics covering expression-level accuracy, relative changes, differential expression (DE) recovery, and distributional similarity. Across tasks, performance depended strongly on perturbation effect size and evaluation perspective: Expression-level agreement was the highest for small-effect perturbations resembling controls, whereas delta- and DE-based metrics improved with larger effects, providing clearer signals. Models shared a conservative bias, with fine-tuned foundation models compressing variance and underestimating synergistic effects in combinations. PerturbNet showed superior recovery of DE signatures in Tasks 1 and 2, while no method consistently generalized across cell types in Task 3, where biological consistency dominated outcomes. This benchmark establishes current methodological limits, clarifies that different metrics probe distinct biological signals rather than redundant summaries of the same prediction problem, and provides a foundation for developing virtual-cell models that more faithfully capture heterogeneous perturbation responses.

Overview

CoChat Summary

TL;DR

This paper presents a large-scale benchmark of 13 single-cell perturbation response prediction methods and baselines across 25 datasets, three prediction tasks, and 24 metrics. It finds that performance depends strongly on perturbation effect size and evaluation metric, with no method consistently generalizing across cell types and with fine-tuned foundation models showing variance compression.

Key findings

  • Across 25 datasets and 24 metrics, absolute expression-level agreement was often high, with Pearson’s correlation exceeding 0.95 for most datasets, but this did not consistently translate into accurate recovery of perturbation-induced changes, DEGs, distributional similarity, or cross-cell-type transfer.
  • PerturbNet showed superior recovery of DE signatures in Tasks 1 and 2, achieving a mean overlap accuracy of 0.56 in Task 1, while most other methods failed to recover DEGs reliably in several datasets.
  • Performance depended strongly on perturbation effect size: small-effect perturbations resembling controls yielded higher expression-level agreement, whereas delta- and DE-based metrics improved with larger effects.
  • In Task 2, simple baselines such as additive mean and linear models were highly competitive and often surpassed complex architectures in seen1 and seen2 combinatorial settings. In Task 3, no method consistently generalized across cell types, and biological consistency dominated outcomes.

Methods

  • Benchmark design: 13 representative methods and baselines were evaluated across three tasks—unseen single-gene perturbation prediction, combinatorial perturbation interaction prediction, and cell state transfer—using 25 datasets spanning genetic perturbations, small-molecule treatments, immune stimuli, cancer cell lines, pluripotent stem cells, primary immune populations, and nonhuman immune cells.
  • Data and models: The collection integrated 23 public datasets and 2 in-house datasets, including two primary immune-cell drug-response resources; models included foundation models (scGPT, scFoundation), GRN-based models (GEARS, scELMo), VAE-based models (PerturbNet, scGen, trVAE, scVIDR, scPRAM), and other deep learning models (CPA, scPreGAN, BioLord, Squidiff), plus task-specific statistical and model-based baselines.
  • Analysis: Evaluation used 24 metrics covering absolute accuracy, relative effect capture, DE recovery, and distributional similarity, with ranking-based meta-analysis and ANOVA variance decomposition to assess contributions of model choice, dataset identity, perturbation identity, effect size, and other factors.

Limitations

  • The paper states that datasets were generated using different technologies, perturbation modalities, cell types, and experimental protocols, introducing batch effects and data-quality variation that can influence training and evaluation; variance analysis indicated intrinsic dataset properties can contribute as much to performance variation as model choice.
  • The paper states that current single-cell sequencing is destructive, so the same cell cannot be observed before and after perturbation; all models infer population-level counterfactuals rather than true paired single-cell trajectories, and the lack of dense temporal sampling limits distinguishing direct perturbation effects from downstream adaptation.
  • The text makes evident that PerturbNet requires chemical structure information (SMILES) and is therefore only applicable to drug perturbation datasets, so not all models were evaluated on all dataset types; Task 3 evaluated only six methods, and DE recovery remained poor across all models in that task.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1126/sciadv.aed3414

Cites

69 of 69 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

11 Valid · 2 Not valid · 0 Not checked · 81 Own results and interpretations

48 more claims · 1 with their cited sources checked

Create a free account to see all 94 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 33
  • Probably not a substantial claim. 12

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 69 references

Cited by

Metadata

Authors
Li, L., You, Y., Fu, Y., Liao, W., Fan, X., Lu, S., Cao, Y., Li, B., Ren, W., Kong, J., Zheng, S., Chen, J., Liu, X., & Tian, L.
Affiliations
  • Guangzhou Laboratory (CN)
  • University of Hong Kong (HK)
  • Westlake University (CN)
  • Renaissance Era Innovation Technology Co., Ltd., Beijing, China.
  • Shanghai Jiao Tong University (CN)
  • First Affiliated Hospital of Guangzhou Medical University (CN)
Corresponding author
  • Li, Lanxiang
  • You, Yue
  • Fu, Yunlin
  • Liu, Xiaodong
  • Tian, Luyi
Venue
Science Advances
Published
2026-09-11
Volume
12
Issue
37
Pages
eaed3414-eaed3414
ISSN
2375-2548 (ISSN-L)
OpenAlex
W7212020435
Cited by
1
References
60
Citations by year
2026: 1
FWCI
2.04
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Science Advances (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
Major Project of Guangzhou National Laboratory (GZNL2024A01015)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Chromatin Dynamics
Keywords
benchmarking, gene expression regulation, drug response, immune cells, foundation models, virtual cell model, evaluation metrics
MeSH terms
Single-Cell Gene Expression Analysis, Animals, Gene Expression Regulation, Humans, Computational Biology, Gene Expression Profiling, Single-Cell Analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.