Simple controls exceed best deep learning algorithms and reveal foundation model effectiveness for predicting genetic perturbations
Open accessPublished version of a preprint
The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
Motivation: Modeling genetic perturbations and their effect on the transcriptome is a key area of pharmaceutical research. Due to the complexity of the transcriptome, there has been much excitement and development in deep learning (DL) because of its ability to model complex relationships. In particular, the transformer-based foundation model paradigm emerged as the gold-standard of predicting post-perturbation responses. However, understanding these increasingly complex models and evaluating their practical utility is lacking, along with simple but appropriate benchmarks to compare predictive methods. Results: Here, we present a simple baseline method that outperforms both state of the art (SOTA) in DL and other proposed simpler neural architectures, setting a necessary benchmark to evaluate in the field of post-perturbation prediction. We also elucidate the utility of foundation models for the task of post-perturbation prediction via generalizable fine-tuning experiments that can be translated to different applications of transformer-based foundation models to tasks of interest. Furthermore, we provide a corrected version of a popular dataset used for benchmarking perturbation prediction models. Our hope is that this work will properly contextualize further development of DL models in the perturbation space with necessary control procedures. Availability and implementation: All source code is available at: https://github.com/pfizer-opensource/perturb_seq. The DOI is 10.5281/zenodo.15352937.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Since the release of our paper as a preprint, several other benchmarks15,16,22–27 were released that also show that deep learning models struggle to outperform simple baselines.”
From Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines (2025) · cites this paper as [27]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
Genome-Scale CRISPR-Mediated Control of Gene Repression and Activation
Gilbert, L. A., Horlbeck, M. A., Adamson, B., Villalta, J. E., Chen, Y., Whitehead, E. H., Guimaraes, C., Panning, B., Ploegh, H. L., Bassik, M. C., et al. (2014) · Cell · doi:10.1016/j.cell.2014.09.029 · Cited by 2,728
Dixit, A., Parnas, O., Li, B., Chen, J., Fulco, C. P., Jerby-Arnon, L., Marjanovic, N. D., Dionne, D., Burks, T., Raychowdhury, R., et al. (2016) · Cell · doi:10.1016/j.cell.2016.11.038 · Cited by 1,863 · cited 2×
“Perturb-seq 4 emerged as a tool to aid this discovery effort by measuring complex post-perturbation transcription profiles.”
scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., & Wang, B. (2023) · doi:10.1101/2023.04.30.538439 · Cited by 1,694
Effective gene expression prediction from sequence by integrating long-range interactions
Avsec, Ž., Agarwal, V., Visentin, D., Ledsam, J. R., Grabska-Barwinska, A., Taylor, K. R., Assael, Y., Jumper, J., Kohli, P., & Kelley, D. R. (2021) · Nature Methods · doi:10.1038/s41592-021-01252-x · Cited by 1,326
CRISPR interference (CRISPRi) for sequence-specific control of gene expression
Larson, M. H., Gilbert, L. A., Wang, X., Lim, W. A., Weissman, J. S., & Qi, L. S. (2013) · Nature Protocols · doi:10.1038/nprot.2013.132 · Cited by 1,212
Adamson, B., Norman, T. M., Jost, M., Cho, M. Y., Nuñez, J. K., Chen, Y., Villalta, J. E., Gilbert, L. A., Horlbeck, M. A., Hein, M. Y., et al. (2016) · Cell · doi:10.1016/j.cell.2016.11.048 · Cited by 1,193 · cited 1×
“Many of these mismatches were the result of cells mislabeled as “ctrl” (i.e. control) in GEARS, but with gene perturbations documented in the cell_identities.csv files provided in the original Adamson et al. paper 45 .”
Transfer learning enables predictions in network biology
Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., Al Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng, Z., Liu, X. S., et al. (2023) · Nature · doi:10.1038/s41586-023-06139-9 · Cited by 1,193 · cited 1×
“As a result of the excitement around foundation models in biology, there has been a large uptake of studies and methods that model single-cell biology via transformers 23,26–31 .”
Mapping information-rich genotype-phenotype landscapes with genome-scale Perturb-seq
Replogle, J. M., Saunders, R. A., Pogson, A. N., Hussmann, J. A., Lenail, A., Guna, A., Mascibroda, L., Wagner, E. J., Adelman, K., Lithwick-Yanai, G., et al. (2022) · Cell · doi:10.1016/j.cell.2022.05.013 · Cited by 795
Harnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X. (2023) · arXiv · doi:10.48550/arxiv.2304.13712 · Cited by 644
Large Scale Foundation Model on Single-cell Transcriptomics
Hao, M., Gong, J., Zeng, X., Liu, C., Guo, Y., Cheng, X., Wang, T., Ma, J., Song, L., & Zhang, X. (2023) · doi:10.1101/2023.05.29.542705 · Cited by 639
Large-scale foundation model on single-cell transcriptomics
Hao, M., Gong, J., Zeng, X., Liu, C., Guo, Y., Cheng, X., Wang, T., Ma, J., Zhang, X., & Song, L. (2024) · Nature Methods · doi:10.1038/s41592-024-02305-7 · Cited by 628 · cited 1×
“Due to the ease of data loading, many studies 27,28,33,43,44 have recently used the ready-made datasets provided by the GEARS 44 repository for training and evaluating models.”
Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., & Yao, J. (2021) · doi:10.1101/2021.12.05.471261 · Cited by 617
A foundation model for clinical-grade computational pathology and rare cancers detection
Vorontsov, E., Bozkurt, A., Casson, A., Shaikovski, G., Zelechowski, M., Severson, K., Zimmermann, E., Hall, J., Tenenholtz, N., Fusi, N., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03141-0 · Cited by 572
Gene regulatory network inference in the era of single-cell multi-omics
Badia-i-Mompel, P., Wessels, L., Müller-Dott, S., Trimbour, R., Ramirez Flores, R. O., Argelaguet, R., & Saez-Rodriguez, J. (2023) · Nature Reviews Genetics · doi:10.1038/s41576-023-00618-5 · Cited by 469
Self-Supervised Representation Learning: Introduction, advances, and challenges
Ericsson, L., Gouk, H., Loy, C. C., & Hospedales, T. M. (2022) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3134634 · Cited by 433
Exploring genetic interaction manifolds constructed from rich single-cell phenotypes
Norman, T. M., Horlbeck, M. A., Replogle, J. M., Ge, A. Y., Xu, A., Jost, M., Gilbert, L. A., & Weissman, J. S. (2019) · Science · doi:10.1126/science.aax4438 · Cited by 392 · cited 8×
“We downloaded the GEARS Adamson, Norman, and Replogle K562 Essential datasets from the Harvard Dataverse using the GEARS interface in June 2024.”
InfluentialPredicting transcriptional outcomes of novel multigene perturbations with GEARS
Roohani, Y., Huang, K., & Leskovec, J. (2024) · Nature Biotechnology · doi:10.1038/s41587-023-01905-6 · Cited by 355 · cited 4×
“For evaluating prediction of the transcriptome for held-out perturbations, Roohani et al. 44 proposed measuring the Pearson correlation between the actual and predicted expression.”
InfluentialThe challenges of in silico biology
Palsson, B. (2000) · Nature Biotechnology · doi:10.1038/81125 · Cited by 306
Ahlmann-Eltze, C., Huber, W., & Anders, S. (2025) · Nature Methods · doi:10.1038/s41592-025-02772-6 · Cited by 238
Transformer Architecture and Attention Mechanisms in Genome Data Analysis: A Comprehensive Review
Choi, S. R., & Lee, M. (2023) · Biology · doi:10.3390/biology12071033 · Cited by 235
Single cells make big data: New challenges and opportunities in transcriptomics
Angerer, P., Simon, L., Tritschler, S., Wolf, F. A., Fischer, D., & Theis, F. J. (2017) · Current Opinion in Systems Biology · doi:10.1016/j.coisb.2017.07.004 · Cited by 221
scPerturb: harmonized single-cell perturbation data
Peidli, S., Green, T. D., Shen, C., Gross, T., Min, J., Garda, S., Yuan, B., Schumacher, L. J., Taylor-King, J. P., Marks, D. S., et al. (2024) · Nature Methods · doi:10.1038/s41592-023-02144-y · Cited by 194
Transformers in single-cell omics: a review and new perspectives
Szałata, A., Hrovatin, K., Becker, S., Tejada-Lapuerta, A., Cui, H., Wang, B., & Theis, F. J. (2024) · Nature Methods · doi:10.1038/s41592-024-02353-z · Cited by 171
Machine learning for perturbational single-cell omics
Ji, Y., Lotfollahi, M., Wolf, F. A., & Theis, F. J. (2021) · Cell Systems · doi:10.1016/j.cels.2021.05.016 · Cited by 137 · cited 1×
“Understanding the phenotypic effects of genetic perturbations is a fundamental area of research in biology 1,2 .”
Cell2Sentence: Teaching Large Language Models the Language of BiologyPreprint
Levine, D. M., Rizvi, S. A., Lévy, S., Pallikkavaliyaveetil, N., Zhang, D., Chen, X., Ghadermarzi, S., Wu, R., Zheng, Z., Vrkic, I., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.09.11.557287 · Cited by 122
Zero-shot evaluation reveals limitations of single-cell foundation models
Kedzierska, K. Z., Crawford, L., Amini, A. P., & Lu, A. X. (2025) · Genome Biology · doi:10.1186/s13059-025-03574-x · Cited by 109
A review of transformer models in drug discovery and beyond
Jiang, J., Chen, L., Ke, L., Dou, B., Zhang, C., Feng, H., Zhu, Y., Qiu, H., Zhang, B., & Wei, G. (2024) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2024.101081 · Cited by 104
Scalable genetic screening for regulatory circuits using compressed Perturb-seq
Yao, D., Binan, L., Bezney, J., Simonton, B., Freedman, J., Frangieh, C. J., Dey, K., Geiger-Schuller, K., Eraslan, B., Gusev, A., et al. (2024) · Nature Biotechnology · doi:10.1038/s41587-023-01964-9 · Cited by 101
Evaluating the Utilities of Foundation Models in Single‐Cell Data Analysis
Liu, T., Li, K., Wang, Y., Li, H., & Zhao, H. (2026) · Advanced Science · doi:10.1002/advs.202514490 · Cited by 99
Evaluating the Utilities of Foundation Models in Single-cell Data Analysis
Liu, T., Li, K., Wang, Y., Li, H., & Zhao, H. (2023) · doi:10.1101/2023.09.08.555192 · Cited by 62
Assessing the limits of zero-shot foundation models in single-cell biology
Kedzierska, K. Z., Crawford, L., Amini, A. P., & Lu, A. X. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.16.561085 · Cited by 61
A mini-review on perturbation modelling across single-cell omic modalities
Gavriilidis, G. I., Vasileiou, V., Orfanou, A., Ishaque, N., & Psomopoulos, F. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.04.058 · Cited by 54
Bhattacharya, M., Pal, S., Chatterjee, S., Lee, S., & Chakraborty, C. (2024) · Molecular Therapy — Nucleic Acids · doi:10.1016/j.omtn.2024.102255 · Cited by 54
A comparison of computational methods for expression forecasting
Kernfeld, E. M., Yang, Y., Weinstock, J. S., Battle, A. J., & Cahan, P. (2025) · Genome biology · doi:10.1186/s13059-025-03840-y · Cited by 50
scMulan: A Multitask Generative Pre-Trained Language Model for Single-Cell Analysis
Bian, H., Chen, Y., Dong, X., Li, C., Hao, M., Chen, S., Hu, J., Sun, M., Wei, L., & Zhang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-1-0716-3989-4_57 · Cited by 46
Data silos are undermining drug development and failing rare disease patients
Denton, N., Molloy, M., Charleston, S., Lipset, C. H., Hirsch, J., Mulberg, A. E., Howard, P., & Marsh, E. D. (2021) · Orphanet Journal of Rare Diseases · doi:10.1186/s13023-021-01806-4 · Cited by 44
A Deep Dive into Single-Cell RNA Sequencing Foundation Models
Boiarsky, R., Singh, N., Buendia, A., Getz, G., & Sontag, D. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.19.563100 · Cited by 43
PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction
Wenteler, A., Occhetta, M., Branson, N., Curean, V., Huebner, M., Dee, W., Connell, W., Chung, S. P., Hawkins-Hooker, A., Ektefaie, Y., et al. (2024) · doi:10.1101/2024.10.02.616248 · Cited by 37
From biobank and data silos into a data commons: convergence to support translational medicine
Asiimwe, R., Lam, S., Leung, S. C. Y., Wang, S., Wan, R., Tinker, A. V., McAlpine, J. N., Woo, M. M. M., Huntsman, D. D., & Talhouk, A. (2021) · Journal of Translational Medicine · doi:10.1186/s12967-021-03147-z · Cited by 36
A Systematic Comparison of Single-Cell Perturbation Response Prediction Models
Li, L., You, Y., Fu, Y., Liao, W., Fan, X., Lu, S., Cao, Y., Li, B., Ren, W., Kong, J., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.12.23.630036 · Cited by 25 · cited 2×
“There have been a growing number of studies in this area 32–40 , but much more analysis needs to be done to fully understand these models.”
Benchmarking AI Models for In Silico Gene Perturbation of Cells
Li, C., Gao, H., She, Y., Bian, H., Chen, Q., Liu, K., Wei, L., & Zhang, X. (2024) · doi:10.1101/2024.12.20.629581 · Cited by 24
A systematic comparison of computational methods for expression forecasting
Kernfeld, E., Yang, Y., Weinstock, J. S., Battle, A., & Cahan, P. (2023) · doi:10.1101/2023.07.28.551039 · Cited by 15 · cited 1×
“Other recent studies have also proposed simple baseline mean variants as competitive methods to more complex regression and even DL-based methods 34–40 .”
Benchmarking a foundational cell model for post-perturbation RNAseq predictionPreprint
Csendes, G., Szalay, K. Z., & Szalai, B. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.30.615843 · Cited by 7
A systematic comparison of single-cell perturbation response prediction models
Li, L., You, Y., Fu, Y., Liao, W., Fan, X., Lu, S., Cao, Y., Li, B., Ren, W., Kong, J., et al. (2026) · Science Advances · doi:10.1126/sciadv.aed3414 · Cited by 1
Research works printed but not linked: 1
- Bian, 2024, Lecture Notes in Computer Science, Lecture Notes in Computer Science, 479
Cited by
Ahlmann-Eltze, C., Huber, W., & Anders, S. (2025) · Nature Methods · doi:10.1038/s41592-025-02772-6 · Cited by 238
TxPert: using multiple knowledge graphs for prediction of transcriptomic perturbation effects
Wenkel, F., Tu, W., Masschelein, C., Shirzad, H., Hodgson, L. M., Bendidi, I., Eastwood, C., Whitfield, S. T., Russell, C., Mesbahi, Y. E., et al. (2026) · Nature Biotechnology · doi:10.1038/s41587-026-03113-4 · Cited by 57
A comparison of computational methods for expression forecasting
Kernfeld, E. M., Yang, Y., Weinstock, J. S., Battle, A. J., & Cahan, P. (2025) · Genome biology · doi:10.1186/s13059-025-03840-y · Cited by 50
Yiu, T., Chen, B., Wang, H., Feng, G., Fu, Q.-Q., & Hu, H. (2025) · Journal of Translational Medicine · doi:10.1186/s12967-025-07091-0 · Cited by 40
A Systematic Comparison of Single-Cell Perturbation Response Prediction Models
Li, L., You, Y., Fu, Y., Liao, W., Fan, X., Lu, S., Cao, Y., Li, B., Ren, W., Kong, J., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.12.23.630036 · Cited by 25
“In single-cell perturbation prediction tasks, baseline models provide reference performance levels that facilitate the evaluation of more expressive predictive models ( 28, 52–54 ).”
Metadata
- Authors
- Wong, D. R., Hill, A. S., & Moccia, R.
- Affiliations
- Pfizer (United States) (US)
- Corresponding author
- Wong, Daniel R
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2025-05-23
- Volume
- 41
- Issue
- 6
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- PMC
- PMC12202205
- OpenAlex
- W4410689365
- Cited by
- 27
- References
- 40
- Citations by year
- 2026: 16 · 2025: 11
- FWCI
- 13.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Bioinformatics (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Pfizer Inc
- Topics
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Keywords
- transcriptomic response, control sequences, baseline model, benchmarking, fine-tuning, Perturb-seq, deep learning evaluation
- MeSH terms
- Deep Learning, Algorithms, Humans, Models, Genetic, Computational Biology, Transcriptome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.