Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index; 14 of the 22 sources its claims cite were checked for support.

Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines

Open access

Published version of a preprint

Abstract

Recent research in deep-learning-based foundation models promises to learn representations of single-cell data that enable prediction of the effects of genetic perturbations. Here we compared five foundation models and two other deep learning models against deliberately simple baselines for predicting transcriptome changes after single or double perturbations. None outperformed the baselines, which highlights the importance of critical benchmarking in directing and evaluating method development.

Overview

CoChat Summary

TL;DR

This benchmark compared five single-cell foundation models and two other deep learning models against deliberately simple baselines for predicting transcriptome changes after single or double gene perturbations. None of the deep learning models consistently outperformed the simple baselines, and a linear model with perturbation embeddings pretrained on Replogle data was the best performer.

Key findings

  • In the double perturbation benchmark on Norman et al. data (100 single genes, 124 gene pairs, 19,264 genes), all models had prediction error substantially higher than the additive baseline, and none was better than the “no change” baseline for genetic interaction prediction. Using the full dataset, the authors identified 5,035 genetic interactions out of potentially 124,000 at a 5% false discovery rate.
  • In the unseen single perturbation benchmark, none of the deep learning models consistently outperformed the mean prediction or the linear model. A linear model with perturbation embeddings pretrained on Replogle data consistently outperformed all other models; gene embeddings from scFoundation and scGPT outperformed the mean baseline but did not consistently outperform the linear model using G and P from the training data.
  • All models mostly predicted buffering interactions, rarely predicted synergistic interactions, and even more rarely predicted them correctly. For most genes, scGPT, UCE and scBERT predictions did not vary across perturbations, and GEARS and scFoundation predictions varied considerably less than the ground truth.

Methods

  • Benchmarked GEARS v0.1.2, scGPT v0.2.1, scFoundation, CPA v0.8.8, Geneformer v0.1.0, scBERT and UCE against simple baselines. Double perturbation analysis used Norman et al. data, training on all 100 single perturbations and 62 double perturbations and testing on the remaining 62 double perturbations across five random splits. Single perturbation analysis used Adamson et al. and Replogle et al. K562/RPE1 datasets with GEARS’ simulation test–training splitting repeated twice.
  • Prediction error was the L2 distance between predicted and observed expression for the 1,000 most highly expressed genes; Pearson delta and other gene subsets were also examined. Genetic interactions were defined as deviations from additive expectation using Efron’s empirical null approach with the locfdr package at a 5% false discovery rate, and true-positive rate versus false discovery proportion curves were computed.
  • The linear model represented read-out genes with matrix G and perturbations with matrix P, found W via normal equations with ridge penalty λ = 0.1, and used K = 10. Baselines included “no change,” “additive” and “mean.” For Geneformer, UCE and scBERT, a ridge regression linear decoder mapped perturbed embeddings to gene expression.

Limitations

  • The paper states that only four datasets were used, chosen because they were used in the publications presenting GEARS, scGPT and scFoundation; all datasets are from cancer cell lines; and the authors did not attempt to improve original quality control, for example by excluding perturbations that did not affect their own target gene.
  • The paper acknowledges that for UCE and scBERT, the post-perturbation embedding was calculated by overwriting rows for perturbed genes with values from the ground truth expression matrix, so test data leakage could theoretically give those models an advantage. It also notes that CPA authors submitted a code fix, but because the new code had worse performance, the original code results are reported.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41592-025-02772-6

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Double perturbation prediction. a, Beeswarm plot of the prediction errors for 62 double perturbations across five test–training splits. The prediction error is measured by the L2 distance between the predicted and the observed expression profile of the n = 1,000 most highly expressed genes. The horizontal red lines show the mean per model, which, for the best-performing model, is extended by the dashed line. b, Scatterplots of observed versus predicted expression from one example of the 62 double perturbations. The numbers indicate error measured by the L2 distance and the Pearson delta (R2). c, TPR (recall) of the interaction predictions as a function of the false discovery proportion. FN, false negative; FP, false positive; TP, true positive. d, Schematic of the classification of interactions based on the difference from the additive expectation (the error bars show the additive range). e, Bar chart of the composition of the observed interaction classes. f, Top: scatterplot of observed versus predicted expression compared to the additive expectation. Each point is one of the 1,000 read-out genes under one of the 62 double perturbations across five test–training splits. The 500 predictions that deviated most from the additive expectation are depicted with bigger and more saturated points. Bottom: mosaic plots that compare the composition of highlighted predictions from the top panel stratified by the interaction class of the prediction. The width of the bars is scaled to match the number of instances. Source data for Fig. 1 are provided. expr., expression. Source data
Fig. 1 Double perturbation prediction. a, Beeswarm plot of the prediction errors for 62 double perturbations across five test–training splits. The prediction error is measured by the L2 distance between the predicted and the observed expression profile of the n = 1,000 most highly expressed genes. The horizontal red lines show the mean per model, which, for the best-performing model, is extended by the dashed line. b, Scatterplots of observed versus predicted expression from one example of the 62 double perturbations. The numbers indicate error measured by the L2 distance and the Pearson delta (R2). c, TPR (recall) of the interaction predictions as a function of the false discovery proportion. FN, false negative; FP, false positive; TP, true positive. d, Schematic of the classification of interactions based on the difference from the additive expectation (the error bars show the additive range). e, Bar chart of the composition of the observed interaction classes. f, Top: scatterplot of observed versus predicted expression compared to the additive expectation. Each point is one of the 1,000 read-out genes under one of the 62 double perturbations across five test–training splits. The 500 predictions that deviated most from the additive expectation are depicted with bigger and more saturated points. Bottom: mosaic plots that compare the composition of highlighted predictions from the top panel stratified by the interaction class of the prediction. The width of the bars is scaled to match the number of instances. Source data for Fig. 1 are provided. expr., expression. Source dataCC BY · PubMed Central
Fig. 2 Single perturbation prediction. a, Beeswarm plot of the prediction errors for 134, 210 and 24 unseen single perturbations across two test–training splits (Methods). The prediction error is measured by the L2 distance between the mean predicted and observed expression profile of the n = 1,000 most highly expressed genes. The horizontal red lines show the mean per model, which, for the best-performing model, is extended by the dashed line. DL, deep learning; LM, linear model. b, Schematic of the LM and how it can accommodate available gene (G) or perturbation (P) embeddings. c, Forest plot comparing the performance of all models relative to the error of the ‘mean’ baseline. The point ranges show the overall mean and 95% confidence interval of the bootstrapped mean ratio between each model and the baseline for 134, 210 and 24 unseen single perturbations across two test–training splits. The opacity of the point range is reduced if the confidence interval contains 0. Source data for Fig. 2 are provided. Source data
Fig. 2 Single perturbation prediction. a, Beeswarm plot of the prediction errors for 134, 210 and 24 unseen single perturbations across two test–training splits (Methods). The prediction error is measured by the L2 distance between the mean predicted and observed expression profile of the n = 1,000 most highly expressed genes. The horizontal red lines show the mean per model, which, for the best-performing model, is extended by the dashed line. DL, deep learning; LM, linear model. b, Schematic of the LM and how it can accommodate available gene (G) or perturbation (P) embeddings. c, Forest plot comparing the performance of all models relative to the error of the ‘mean’ baseline. The point ranges show the overall mean and 95% confidence interval of the bootstrapped mean ratio between each model and the baseline for 134, 210 and 24 unseen single perturbations across two test–training splits. The opacity of the point range is reduced if the confidence interval contains 0. Source data for Fig. 2 are provided. Source dataCC BY · PubMed Central
Extended Data Fig. 1 Dataset Overview. (a) Table with the size of the data and the number of perturbations. (b) UMAP on the perturbations per dataset (aggregated to the mean per perturbation). The position of the control condition without perturbation is shown in red, and a random selection of perturbations is labeled. (c) Change in the expression of the target gene of each perturbation. The base of the arrow indicates the expression without perturbation, and the tip indicates the expression after perturbation. For genes targeted multiple times in the Norman dataset, we show the average expression after perturbation. Source data
Extended Data Fig. 1 Dataset Overview. (a) Table with the size of the data and the number of perturbations. (b) UMAP on the perturbations per dataset (aggregated to the mean per perturbation). The position of the control condition without perturbation is shown in red, and a random selection of perturbations is labeled. (c) Change in the expression of the target gene of each perturbation. The base of the arrow indicates the expression without perturbation, and the tip indicates the expression after perturbation. For genes targeted multiple times in the Norman dataset, we show the average expression after perturbation. Source dataCC BY · PubMed Central
Extended Data Fig. 2 Alternative measures of the double perturbation prediction performance. (a) The Pearson delta measure calculates the correlation of the prediction and observations after subtracting the expression in the control condition. The correlation for the no change predictions could not be calculated because they were all zero. The horizontal red lines show the mean per model, and the dashed line indicates the correlation of the best-performing model. (b) Prediction error as a function of n, the number of read-out genes. Left: genes ranked by expression in the control condition, right: by differential expression between observed value and expression in the control condition. Note that sorting by differential expression is only possible if access to the ground truth is available and can thus not be applied in real-world use cases. The dashed line at n = 1000 is the choice in Panel a and elsewhere in this work. Source data
Extended Data Fig. 2 Alternative measures of the double perturbation prediction performance. (a) The Pearson delta measure calculates the correlation of the prediction and observations after subtracting the expression in the control condition. The correlation for the no change predictions could not be calculated because they were all zero. The horizontal red lines show the mean per model, and the dashed line indicates the correlation of the best-performing model. (b) Prediction error as a function of n, the number of read-out genes. Left: genes ranked by expression in the control condition, right: by differential expression between observed value and expression in the control condition. Note that sorting by differential expression is only possible if access to the ground truth is available and can thus not be applied in real-world use cases. The dashed line at n = 1000 is the choice in Panel a and elsewhere in this work. Source dataCC BY · PubMed Central
Extended Data Fig. 3 Distribution of the observed difference from the additive model. (a) Quantile-quantile plot comparing the distribution of the differences between observed expression values and the additive expectation against a standard normal distribution. The slope of the line is the standard deviation of the null model. (b) Histogram of the differences with a red curve overlayed that shows the null distribution fitted using locfdr. Values under the curve are grey, and the black bars show the observations that exceed what we would expect under the null model. The vertical bar shows the upper and lower thresholds for which the observations have a false discovery rate of less than 5% (that is, the grey fraction of the bars outside the vertical lines is 5%). The numbers at the top count the observations per group. Source data
Extended Data Fig. 3 Distribution of the observed difference from the additive model. (a) Quantile-quantile plot comparing the distribution of the differences between observed expression values and the additive expectation against a standard normal distribution. The slope of the line is the standard deviation of the null model. (b) Histogram of the differences with a red curve overlayed that shows the null distribution fitted using locfdr. Values under the curve are grey, and the black bars show the observations that exceed what we would expect under the null model. The vertical bar shows the upper and lower thresholds for which the observations have a false discovery rate of less than 5% (that is, the grey fraction of the bars outside the vertical lines is 5%). The numbers at the top count the observations per group. Source dataCC BY · PubMed Central
Extended Data Fig. 4 Alternative measures how well each model detects genetic interactions. (a) Precision-recall and (b) receiver operator curve for all models distinguishing interactions from additive combinations. The numbers in parenthesis are the area under the curve (AUC) with the standard error across five test-training splits. TP: true positive, FP: false positive, FN: false negative, TN: true negative.
Extended Data Fig. 4 Alternative measures how well each model detects genetic interactions. (a) Precision-recall and (b) receiver operator curve for all models distinguishing interactions from additive combinations. The numbers in parenthesis are the area under the curve (AUC) with the standard error across five test-training splits. TP: true positive, FP: false positive, FN: false negative, TN: true negative.CC BY · PubMed Central
Extended Data Fig. 5 Reoccurrence of genes and perturbations for top predictions. (a) Reoccurrence of genes and (b) perturbations among the 100 predictions that differed most from the additive expectation. The data is facetted by the test-training split. The ground truth column shows the genes and perturbations sorted by observed difference from the additive expectation. The highlighted genes and perturbations are the four most reoccurring ones. Source data
Extended Data Fig. 5 Reoccurrence of genes and perturbations for top predictions. (a) Reoccurrence of genes and (b) perturbations among the 100 predictions that differed most from the additive expectation. The data is facetted by the test-training split. The ground truth column shows the genes and perturbations sorted by observed difference from the additive expectation. The highlighted genes and perturbations are the four most reoccurring ones. Source dataCC BY · PubMed Central
Extended Data Fig. 6 Comparison of predicted and observed expression for HBG2 and HBZ. Comparison of the predicted expression (black squares), the observed expression values (points colored by interaction type), and the range of values that are considered additive (grey boxes) for all test perturbations with seed = 1. The grey horizontal line shows the expression of HBG2 and HBZ without perturbation. Source data
Extended Data Fig. 6 Comparison of predicted and observed expression for HBG2 and HBZ. Comparison of the predicted expression (black squares), the observed expression values (points colored by interaction type), and the range of values that are considered additive (grey boxes) for all test perturbations with seed = 1. The grey horizontal line shows the expression of HBG2 and HBZ without perturbation. Source dataCC BY · PubMed Central
Extended Data Fig. 7 Variation of the predicted and observed expression values. Histogram of the standard deviation per gene for the predicted and observed expression values across perturbations facetted by the model. The red vertical bar indicates the mean of the standard deviations for the ground truth for (a) the Norman dataset and (b) the Replogle K562 dataset. The data reflects the variation for the 1 000 most highly expressed genes and is aggregated across five test-training splits. LM: linear model.
Extended Data Fig. 7 Variation of the predicted and observed expression values. Histogram of the standard deviation per gene for the predicted and observed expression values across perturbations facetted by the model. The red vertical bar indicates the mean of the standard deviations for the ground truth for (a) the Norman dataset and (b) the Replogle K562 dataset. The data reflects the variation for the 1 000 most highly expressed genes and is aggregated across five test-training splits. LM: linear model.CC BY · PubMed Central
Extended Data Fig. 8 Alternative measures of the single perturbation prediction performance. (a) The Pearson delta measure calculates the correlation of the prediction and observations after subtracting the expression in the control condition. The horizontal red lines show the mean per model and the dashed line indicates the correlation of the best-performing model. (b) Prediction error as a function of n, the number of read-out genes. Top: genes ranked by expression in the control condition. Bottom: by differential expression between observed value and expression in the control condition. Note that sorting by differential expression is only possible if access to the ground truth is available and can thus not be applied in real-world use cases. The dashed line at n = 1000 is the choice in Panel a and elsewhere in this work. LM: linear model, DL: deep learning. Source data
Extended Data Fig. 8 Alternative measures of the single perturbation prediction performance. (a) The Pearson delta measure calculates the correlation of the prediction and observations after subtracting the expression in the control condition. The horizontal red lines show the mean per model and the dashed line indicates the correlation of the best-performing model. (b) Prediction error as a function of n, the number of read-out genes. Top: genes ranked by expression in the control condition. Bottom: by differential expression between observed value and expression in the control condition. Note that sorting by differential expression is only possible if access to the ground truth is available and can thus not be applied in real-world use cases. The dashed line at n = 1000 is the choice in Panel a and elsewhere in this work. LM: linear model, DL: deep learning. Source dataCC BY · PubMed Central
Extended Data Fig. 9 Analysis how differential expression between K562 and RPE1 effects prediction accuracy of transfer learning. (a) Scatter plot of the mean gene expression for shared genes between RPE1 and K562 without perturbation. The dashed line indicates the diagonal. (b) Scatter plot of the absolute prediction error per read-out gene against the differential expression of that gene between RPE1 and K562. Each point is one read-out gene from one of the 122 double perturbations from five test-training splits. The blue line shows the linear fit with a slope indicated in the subtitle. (c) Scatter plot of the Pearson delta score per perturbation for the RPE1 dataset against the differential expression of the perturbation target gene between RPE1 and K562. The blue line shows the linear fit with a slope indicated in the subtitle, and the shaded area indicates the standard error of the fit. Source data
Extended Data Fig. 9 Analysis how differential expression between K562 and RPE1 effects prediction accuracy of transfer learning. (a) Scatter plot of the mean gene expression for shared genes between RPE1 and K562 without perturbation. The dashed line indicates the diagonal. (b) Scatter plot of the absolute prediction error per read-out gene against the differential expression of that gene between RPE1 and K562. Each point is one read-out gene from one of the 122 double perturbations from five test-training splits. The blue line shows the linear fit with a slope indicated in the subtitle. (c) Scatter plot of the Pearson delta score per perturbation for the RPE1 dataset against the differential expression of the perturbation target gene between RPE1 and K562. The blue line shows the linear fit with a slope indicated in the subtitle, and the shaded area indicates the standard error of the fit. Source dataCC BY · PubMed Central
Extended Data Fig. 10 Computational resource requirements. The resource usage was measured for the Norman dataset, which had 19 624 genes and 81 143 cells grouped into 225 conditions. Each point is one of the five test-training splits. (a) Elapsed time on a log scale to fine-tune and predict the double perturbations. (b) Peak memory usage for each model was measured using GNU time. The points are colored by the respective GPU model that was used. Source data
Extended Data Fig. 10 Computational resource requirements. The resource usage was measured for the Norman dataset, which had 19 624 genes and 81 143 cells grouped into 225 conditions. Each point is one of the five test-training splits. (a) Elapsed time on a log scale to fine-tune and predict the double perturbations. (b) Peak memory usage for each model was measured using GNU time. The points are colored by the respective GPU model that was used. Source dataCC BY · PubMed Central

Verification

Claims the paper makes

6 Valid · 0 Not valid · 5 Not checked · 32 Own results and interpretations

22 more claims

Create a free account to see all 43 claims and to verify the 2 references still open.

See all claims — free account
  • Probably not a substantial claim. 15
  • It may state two separate claims as one. 3

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 31 references

Research works printed but not linked: 1

  1. Märtens, K., Donovan-Maiye, R. & Ferkinghoff-Borg, J. Enhancing generative perturbation models with LLM-informed gene embeddings. In Proc. Workshop on Machine Learning for Genomics Explorations (ICLR, 2024). https://openreview.net/forum?id=eb3ndUlkt4

Cited by

See all 238

Metadata

Authors
Ahlmann-Eltze, C., Huber, W., & Anders, S.
Affiliations
  • Heidelberg University (DE)
  • European Molecular Biology Laboratory (DE)
Corresponding author
Ahlmann-Eltze, Constantin
Venue
Nature Methods
Published
2025-08
Volume
22
Issue
8
Pages
1657-1661
ISSN
  • 1548-7091 (ISSN-L)
  • 1548-7105
OpenAlex
W4412929241
Cited by
238
References
31
Citations by year
2026: 147 · 2025: 78
FWCI
67.16
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Methods (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
Klaus Tschira Stiftung (00.022.2019)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
Keywords
single-cell transcriptomics, foundation models, deep learning, benchmarking, transcriptome changes
MeSH terms
Deep Learning, Algorithms, Humans, Computational Biology, Gene Expression Profiling, Single-Cell Analysis, Transcriptome

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines. https://app.cochat.ai/papers/10.1038/s41592-025-02772-6
The paper
Ahlmann-Eltze, C., Huber, W., & Anders, S. (2025). Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines. Nature Methods. https://doi.org/10.1038/s41592-025-02772-6

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.