CoChat

Simple controls exceed best deep learning algorithms and reveal foundation model effectiveness for predicting genetic perturbations

Open access
Wong, Daniel R, Hill, Abby S, Moccia, Rob · 2025 · Bioinformatics

Published version of a preprint

The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.

Abstract

Motivation: Modeling genetic perturbations and their effect on the transcriptome is a key area of pharmaceutical research. Due to the complexity of the transcriptome, there has been much excitement and development in deep learning (DL) because of its ability to model complex relationships. In particular, the transformer-based foundation model paradigm emerged as the gold-standard of predicting post-perturbation responses. However, understanding these increasingly complex models and evaluating their practical utility is lacking, along with simple but appropriate benchmarks to compare predictive methods. Results: Here, we present a simple baseline method that outperforms both state of the art (SOTA) in DL and other proposed simpler neural architectures, setting a necessary benchmark to evaluate in the field of post-perturbation prediction. We also elucidate the utility of foundation models for the task of post-perturbation prediction via generalizable fine-tuning experiments that can be translated to different applications of transformer-based foundation models to tasks of interest. Furthermore, we provide a corrected version of a popular dataset used for benchmarking perturbation prediction models. Our hope is that this work will properly contextualize further development of DL models in the perturbation space with necessary control procedures. Availability and implementation: All source code is available at: https://github.com/pfizer-opensource/perturb_seq. The DOI is 10.5281/zenodo.15352937.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/bioinformatics/btaf317

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. CRISPR-informed mean as SOTA predictor. (a) Performance bar charts on held-out test set for different models. Error bars span plus and minus one standard deviation of 10 independently trained models applied to the same test set. (b) Boxplots of top 20 DE genes for a subset of scGPT’s high performing held-out perturbation conditions for each dataset. Target gene T is boxed in red when T is among the top 20 DE genes. The title of each plot denotes the dataset, perturbation condition, and scGPT’s Pearson DE score for predicting the expression of cellstarget=T. In the figure legend “T≠condition” denotes all perturbed cellstarget≠T in the dataset. “T=condition” denotes all cellstarget=T. (c) Boxplots of Pearson correlations between average expression of cellstarget=T and average expression of perturbed cellstarget≠T repeated over each target T in the whole dataset. Pearson correlation was calculated over all genes for all perturbations in the dataset. (d) Boxplots of Wasserstein distances. Left: boxplots of Wasserstein distances between (1) expression of T in cellstarget=T and (2) expression of T in perturbed cellstarget≠T. Right: boxplots of Wasserstein distances ∀ gene D≠T ∈ DE genes between (1) expression of D in cellstarget=T and (2) expression of D in perturbed cellstarget≠T. Boxplots are computed over all perturbations in the dataset. The differences between the left and right distributions were all statistically significant with the following P-values for a two-sided t-test: Corrected Adamson UPR = 3.6 × 10−13, Norman = 4.5 × 10−5, Replogle K562 Essential = 4.3 × 10−173. (e) Rank comparisons between different methods over the held-out perturbation test set. A rank of 0.0 indicates a perfect score, and a rank of 0.5 is the expected score from a random prediction. For held-out perturbation with target = T, the reference gene expression used to compute rank was the average of the ground truth expression of cellstarget=T over all genes. Since the prediction of the mean model is the same for all perturbations, the cosine similarity to reference is the same for all perturbations and results in a random rank. For each model type, we used the best-performing model from the ten independently trained models that achieved the highest PDED performance on the test set.
Figure 1. CRISPR-informed mean as SOTA predictor. (a) Performance bar charts on held-out test set for different models. Error bars span plus and minus one standard deviation of 10 independently trained models applied to the same test set. (b) Boxplots of top 20 DE genes for a subset of scGPT’s high performing held-out perturbation conditions for each dataset. Target gene T is boxed in red when T is among the top 20 DE genes. The title of each plot denotes the dataset, perturbation condition, and scGPT’s Pearson DE score for predicting the expression of cellstarget=T. In the figure legend “T≠condition” denotes all perturbed cellstarget≠T in the dataset. “T=condition” denotes all cellstarget=T. (c) Boxplots of Pearson correlations between average expression of cellstarget=T and average expression of perturbed cellstarget≠T repeated over each target T in the whole dataset. Pearson correlation was calculated over all genes for all perturbations in the dataset. (d) Boxplots of Wasserstein distances. Left: boxplots of Wasserstein distances between (1) expression of T in cellstarget=T and (2) expression of T in perturbed cellstarget≠T. Right: boxplots of Wasserstein distances ∀ gene D≠T ∈ DE genes between (1) expression of D in cellstarget=T and (2) expression of D in perturbed cellstarget≠T. Boxplots are computed over all perturbations in the dataset. The differences between the left and right distributions were all statistically significant with the following P-values for a two-sided t-test: Corrected Adamson UPR = 3.6 × 10−13, Norman = 4.5 × 10−5, Replogle K562 Essential = 4.3 × 10−173. (e) Rank comparisons between different methods over the held-out perturbation test set. A rank of 0.0 indicates a perfect score, and a rank of 0.5 is the expected score from a random prediction. For held-out perturbation with target = T, the reference gene expression used to compute rank was the average of the ground truth expression of cellstarget=T over all genes. Since the prediction of the mean model is the same for all perturbations, the cosine similarity to reference is the same for all perturbations and results in a random rank. For each model type, we used the best-performing model from the ten independently trained models that achieved the highest PDED performance on the test set.CC BY · PubMed Central
Figure 2. Performance of weight and architecture permutations. (a) scGPT model architecture for the Perturb-seq task. Here we denote 3 different ways to selectively use the foundation weights. (1) No-pretraining, ie not using them at all and initializing all weights randomly (2) loading the foundation input encoder weights and initializing all other weights randomly and (3) loading the foundation transformer encoder weights and initializing all other weights randomly. (b) Performance bar charts on held-out test set for different baseline models. Error bars span plus and minus one standard deviation of 10 independently trained models.
Figure 2. Performance of weight and architecture permutations. (a) scGPT model architecture for the Perturb-seq task. Here we denote 3 different ways to selectively use the foundation weights. (1) No-pretraining, ie not using them at all and initializing all weights randomly (2) loading the foundation input encoder weights and initializing all other weights randomly and (3) loading the foundation transformer encoder weights and initializing all other weights randomly. (b) Performance bar charts on held-out test set for different baseline models. Error bars span plus and minus one standard deviation of 10 independently trained models.CC BY · PubMed Central
Figure 3. Simple Affine model is competitive with transformer-based model. (a) Architecture of the Simple Affine model. (b) Performance bar charts on held-out test set for different baseline models. Error bars span plus and minus one standard deviation of 10 independently trained models. (c) Bar charts comparing Simple Affine and scGPT average training time per epoch for each dataset on one Nvidia H100 GPU. (d) Bar charts comparing Simple Affine and scGPT model sizes.
Figure 3. Simple Affine model is competitive with transformer-based model. (a) Architecture of the Simple Affine model. (b) Performance bar charts on held-out test set for different baseline models. Error bars span plus and minus one standard deviation of 10 independently trained models. (c) Bar charts comparing Simple Affine and scGPT average training time per epoch for each dataset on one Nvidia H100 GPU. (d) Bar charts comparing Simple Affine and scGPT model sizes.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 1

  1. Bian, 2024, Lecture Notes in Computer Science, Lecture Notes in Computer Science, 479

Cited by

See all 42

Metadata

Authors
Wong, D. R., Hill, A. S., & Moccia, R.
Affiliations
Pfizer (United States) (US)
Corresponding author
Wong, Daniel R
Venue
Bioinformatics
Published
2025-05-23
Volume
41
Issue
6
ISSN
  • 1367-4803 (ISSN-L)
  • 1367-4811
OpenAlex
W4410689365
Cited by
27
References
40
Citations by year
2026: 16 · 2025: 11
FWCI
13.95
Type
journal article
Language
en
Open access
Locations
  • Bioinformatics (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
Pfizer Inc
Topics
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
Keywords
transcriptomic response, control sequences, baseline model, benchmarking, fine-tuning, Perturb-seq, deep learning evaluation
MeSH terms
Deep Learning, Algorithms, Humans, Models, Genetic, Computational Biology, Transcriptome

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.