Genetic Perturbation Prediction Models Fail Out-of-Distribution: A Rigorous Benchmark and Systematic Evaluation
Open accessAbstract
Perturbation prediction models - which forecast transcriptional responses to gene inactivation - are increasingly proposed as in silico surrogates for genome-scale drug discovery screens. Yet, rigorous evaluation of whether these models genuinely generalize beyond their training distribution has been lacking. Existing benchmarks rely predominantly on random train-test splits, obscuring systematic failure modes that only emerge under realistic out-of-distribution (OOD) conditions. Here we introduce EL-PERTURB (Extrapolative Learning for PERTURBation Prediction), a benchmarking framework that stress-tests 20 perturbation prediction models spanning six architecture classes on both essential gene and genome-wide Perturb-seq screens, using a novel stratified OOD splitting strategy that partitions train and test sets into functionally distinct regions of transcriptional space. We find that model performance degrades substantially under OOD conditions, and that architectural complexity - including single-cell foundation model embeddings - confers no reliable advantage over well-tuned simpler baselines. EL-PERTURB establishes a rigorous benchmark for perturbation effect prediction across a gradient of generalization difficulty, providing a guide for building models that can genuinely serve as surrogates for wet-lab screening.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Gene set knowledge discovery with enrichr
Xie, Z., Bailey, A., Kuleshov, M. V., Clarke, D. J. B., Evangelista, J. E., Jenkins, S. L., Lachmann, A., Wojciechowicz, M. L., Kropiwnicki, E., Jagodnik, K. M., et al. (2021) · Current Protocols · doi:10.1002/cpz1.90 · Cited by 3,451 · see citing works
Film: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490 · see citing works
scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., & Wang, B. (2023) · doi:10.1101/2023.04.30.538439 · Cited by 1,383 · see citing works
Transfer learning enables predictions in network biology
Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., Al Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng, Z., Liu, X. S., et al. (2023) · Nature · doi:10.1038/s41586-023-06139-9 · Cited by 1,193 · see citing works
Mapping information-rich genotype-phenotype landscapes with genome-scale Perturb-seq
Replogle, J. M., Saunders, R. A., Pogson, A. N., Hussmann, J. A., Lenail, A., Guna, A., Mascibroda, L., Wagner, E. J., Adelman, K., Lithwick-Yanai, G., et al. (2022) · Cell · doi:10.1016/j.cell.2022.05.013 · Cited by 751 · see citing works
scGen predicts single-cell perturbation responses
Lotfollahi, M., Wolf, F. A., & Theis, F. J. (2019) · Nature Methods · doi:10.1038/s41592-019-0494-8 · Cited by 661 · see citing works
How to build the virtual cell with artificial intelligence: Priorities and opportunities
Bunne, C., Roohani, Y., Rosen, Y., Gupta, A., Zhang, X., Roed, M., Alexandrov, T., AlQuraishi, M., Brennan, P., Burkhardt, D. B., et al. (2024) · Cell · doi:10.1016/j.cell.2024.11.015 · Cited by 393 · see citing works
Predicting transcriptional outcomes of novel multigene perturbations with GEARS
Roohani, Y., Huang, K., & Leskovec, J. (2024) · Nature Biotechnology · doi:10.1038/s41587-023-01905-6 · Cited by 355 · see citing works
Predicting cellular responses to complex perturbations in high‐throughput screens
Lotfollahi, M., Klimovskaia Susmelj, A., De Donno, C., Hetzel, L., Ji, Y., Ibarra, I. L., Srivatsan, S. R., Naghipourfar, M., Daza, R. M., Martin, B., et al. (2023) · Molecular Systems Biology · doi:10.15252/msb.202211517 · Cited by 307 · see citing works
Ahlmann-Eltze, C., Huber, W., & Anders, S. (2025) · Nature Methods · doi:10.1038/s41592-025-02772-6 · Cited by 238 · see citing works
scPerturb: harmonized single-cell perturbation data
Peidli, S., Green, T. D., Shen, C., Gross, T., Min, J., Garda, S., Yuan, B., Schumacher, L. J., Taylor-King, J. P., Marks, D. S., et al. (2024) · Nature Methods · doi:10.1038/s41592-023-02144-y · Cited by 194 · see citing works
Zero-shot evaluation reveals limitations of single-cell foundation models
Kedzierska, K. Z., Crawford, L., Amini, A. P., & Lu, A. X. (2025) · Genome Biology · doi:10.1186/s13059-025-03574-x · Cited by 109 · see citing works
Predicting cellular responses to perturbation across diverse contexts with State
Adduri, A. K., Gautam, D., Bevilacqua, B., et al. (2025) · doi:10.1101/2025.06.26.661135 · Cited by 107
Predicting cellular responses to perturbation across diverse contexts with State
Adduri, A. K., Gautam, D., Bevilacqua, B., Naghipourfar, M., Imran, A., Shah, R., Teyssier, N., Verma, R., Carpenter, C., Eraslan, B., et al. (2026) · Cell · doi:10.1016/j.cell.2026.07.052 · Cited by 100 · see citing works
Virtual Cell Challenge: Toward a Turing test for the virtual cell
Roohani, Y., Hua, T. J., Tung, P., Bounds, L. R., Yu, F. B., Dobin, A., Teyssier, N., Adduri, A., Woodrow, A., Plosky, B. S., et al. (2025) · Cell · doi:10.1016/j.cell.2025.06.008 · Cited by 95 · see citing works
Transcriptome-wide analysis of differential expression in perturbation atlases
Nadig, A., Replogle, J. M., Pogson, A. N., Murthy, M., McCarroll, S. A., Weissman, J. S., Robinson, E. B., & O’Connor, L. J. (2025) · Nature Genetics · doi:10.1038/s41588-025-02169-3 · Cited by 70 · see citing works
Viñas Torné, R., Wiatrak, M., Piran, Z., Fan, S., Jiang, L., Teichmann, S. A., Nitzan, M., & Brbić, M. (2026) · Nature Biotechnology · doi:10.1038/s41587-025-02777-8 · Cited by 58 · see citing works
Ma, C., Zhang, H., Rao, Y., Jiang, X., Liu, B., Sun, Z., Song, Z., Gao, Y., Cui, Y., Liu, X., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02198-6 · Cited by 52 · see citing works
Huang, A. C., Hsieh, T.-H. S., Zhu, J., Michuda, J., Teng, A., Kim, S., Rumsey, E. M., Lam, S. K., Anigbogu, I., Wright, P., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.06.11.659105 · Cited by 50 · see citing works
Benchmarking foundation cell models for post-perturbation RNA-seq prediction
Csendes, G., Sanz, G., Szalay, K. Z., & Szalai, B. (2025) · BMC Genomics · doi:10.1186/s12864-025-11600-2 · Cited by 47 · see citing works
Benchmarking algorithms for generalizable single-cell perturbation response prediction
Wei, Z., Wang, Y., Gao, Y., Wang, S., Li, P., Si, D., Gao, Y., Wu, S., Li, D., Dong, K., et al. (2026) · Nature Methods · doi:10.1038/s41592-025-02980-0 · Cited by 42 · see citing works
Zhu, R., Dann, E., Yan, J., Retana, J. R., Goto, R., Guitche, R. C., Brixi, L., Ota, M., Hartman, A., Roth, T. L., et al. (2026) · Cell · doi:10.1016/j.cell.2026.08.002 · Cited by 35 · see citing works
Modeling and predicting single-cell multi-gene perturbation responses with scLAMBDAPreprint
Wang, G., Liu, T., Zhao, J., Cheng, Y., & Zhao, H. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.12.04.626878 · Cited by 24 · see citing works
TxPert: using multiple knowledge graphs for prediction of transcriptomic perturbation effects.
Wenkel, F., Tu, W., Masschelein, C., Shirzad, H., Hodgson, L., Bendidi, I., Eastwood, C., Whitfield, S. T., Russell, C., Mesbahi, Y. E., et al. (2026) · Nature Biotechnology · doi:10.1038/s41587-026-03113-4 · Cited by 23 · see citing works
Szalai, B., & Veres, D. V. (2023) · Frontiers in Systems Biology · doi:10.3389/fsysb.2023.1126044 · Cited by 21 · see citing works
GenePert: Leveraging GenePT Embeddings for Gene Perturbation PredictionPreprint
Chen, Y. T., & Zou, J. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.27.620513 · Cited by 21 · see citing works
PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis
Wu, Y., Wershof, E., Schmon, S., Nassar, M., Osiński, B., Eksi, R., Yan, Z., Stark, R., Zhang, K., & Graepel, T. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-3225 · Cited by 20 · see citing works
Littman, R., Levine, J., Maleki, S., Lee, Y., Ermakov, V. Y., Qiu, L., Wu, A. P., Huang, K., Lopez, R., Scalia, G., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.06.03.657653 · Cited by 10 · see citing works
(2024) · Nature methods · doi:10.1038/s41592-024-02201-0
Unresolved- Unresolved
Ne- manja D Marjanovic, Danielle Dionne, Tyler Burks, Raktima Raychowdhury
(2016) · Perturb-seq: dissecting molecular circuits with scalable single-cell rna profiling of pooled genetic screens. Cell
UnresolvedGabriel M Mejia , Henry E Miller , Francis JA Leblanc , Bo Wang , Brendan Swain , and Lucas Paulo de Lima Camillo. Diversity by design: Addressing mode collapse improves scrna-seq perturbation modeling on well-calibrated metrics. arXiv preprint arXiv:2506.22641, 2025.
(n.d.)
UnresolvedIhab Bendidi , Shawn Whitfield , Kian Kenyon-Dean , Hanene Ben Yedder , Yassir El Mesbahi , Emmanuel Noutahi , and Alisandra K Denton . Benchmarking transcriptomics foundation models for perturbation analysis: one pca still rules them all. arXiv preprint arXiv:2410.13956, 2024.
(n.d.)
UnresolvedKaspar Märtens , Marc Boubnovski Martell , Cesar A Prada-Medina , and Rory Donovan-Maiye . Langpert: Llm-driven contextual synthesis for unseen perturbation prediction. Openreview.net, 2025.
(n.d.)
Unresolved
Flagged first, then by citations
6 of the 31 references couldn’t be matched to a known paper.
- Authors
- Tiwari, K., Mahajan, G., Bellad, R., Dhruw, N. K., & Jha, A.
- Affiliations
- Elucidata Corporation, 2261 Market Street, STE 10296, San Francisco, CA 94114
- Corresponding author
- Jha, Abhishek
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2026-10-19
- OpenAlex
- W7222954140
- Cited by
- 0
- References
- 31
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Keywords
- perturbation prediction, gene inactivation, transcriptional response, out-of-distribution generalization, benchmarking framework, Perturb-seq, essential genes, single-cell foundation models, model architecture comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
