Systema: a framework for evaluating genetic perturbation response prediction beyond systematic variation
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
Predicting transcriptional responses to genetic perturbations is challenging in functional genomics. While recent methods aim to infer effects of untested perturbations, their true predictive power remains unclear. Here, we show that current methods struggle to generalize beyond systematic variation, the consistent transcriptional differences between perturbed and control cells arising from selection biases or confounders. We quantify this variation in ten datasets, spanning three technologies and five cell lines, and show that common metrics are susceptible to these biases, leading to overestimated performance. To address this, we introduce Systema, an evaluation framework that emphasizes perturbation-specific effects and identifies predictions that correctly reconstruct the perturbation landscape. Using this framework, we uncover insights into the predictive capabilities of existing methods and show that predicting responses to unseen perturbations is substantially harder than standard metrics suggest. Our work highlights the importance of heterogeneous gene panels and disentangles predictive performance from systematic effects, enabling biologically meaningful developments in perturbation response modeling.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While this work provides valuable insights, its evaluation was limited in dataset diversity and task coverage, leaving the question of whether such conclusions generalize across broader biological contexts.”
From A systematic comparison of single-cell perturbation response prediction models (2026) · cites this paper as [26]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These observations are consistent with recent work showing that standard perturbation metrics can overestimate biological prediction accuracy and that metric design remains a critical challenge for virtual-cell benchmarking (26, 30, 31).”
From A systematic comparison of single-cell perturbation response prediction models (2026) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Recent studies have provided initial benchmarks comparing foundation models, other deep learning architectures, and simple baselines for perturbation effect prediction and reported that complex models do not consistently outperform the simpler alternatives (22–26).”
From A systematic comparison of single-cell perturbation response prediction models (2026) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee, S., Ebert, B. L., Gillette, M. A., Paulovich, A., Pomeroy, S. L., Golub, T. R., Lander, E. S., et al. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0506580102 · Cited by 53,625 · cited 5×
“To identify enriched pathways (between perturbed and control cells), we performed GSEA on the processed data using GSEApy 30 and Biological Process 2023 gene sets from GO.”
InfluentialGene Ontology: tool for the unification of biology
Ashburner, M., Ball, C. A., Blake, J. A., Botstein, D., Butler, H., Cherry, J. M., Davis, A. P., Dolinski, K., Dwight, S. S., Eppig, J. T., et al. (2000) · Nature Genetics · doi:10.1038/75556 · Cited by 43,206
SCANPY: large-scale single-cell gene expression data analysis
Wolf, F. A., Angerer, P., & Theis, F. J. (2018) · Genome Biology · doi:10.1186/s13059-017-1382-0 · Cited by 8,881 · cited 1×
“27–32 and Supplementary Tables 3–6).”
Scenic: single-cell regulatory network inference and clustering.
Aibar, S., González-Blas, C. B., Moerman, T., Huynh-Thu, V. A., Imrichova, H., Hulselmans, G., Rambow, F., Marine, J.-C., Geurts, P., Aerts, J., et al. (2017) · Nature Methods · doi:10.1038/nmeth.4463 · Cited by 6,520
Dissecting the multicellular ecosystem of metastatic melanoma by single-cell RNA-seq
Tirosh, I., Izar, B., Prakadan, S. M., Wadsworth, M. H., Treacy, D., Trombetta, J. J., Rotem, A., Rodman, C., Lian, C., Murphy, G., et al. (2016) · Science · doi:10.1126/science.aad0501 · Cited by 4,944 · cited 2×
“33) and results were robust to the panel of genetic perturbations (Supplementary Fig.”
The Gene Ontology knowledgebase in 2023
The Gene Ontology Consortium, Aleksander, S. A., Balhoff, J., Carbon, S., Cherry, J. M., Drabkin, H. J., Ebert, D., Feuermann, M., Gaudet, P., Harris, N. L., et al. (2023) · GENETICS · doi:10.1093/genetics/iyad031 · Cited by 2,664
scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., & Wang, B. (2023) · doi:10.1101/2023.04.30.538439 · Cited by 1,694
RNA virus interference via CRISPR/Cas13a system in plants
Aman, R., Ali, Z., Butt, H., Mahas, A., Aljedaani, F., Khan, M. Z., Ding, S., & Mahfouz, M. (2018) · Genome Biology · doi:10.1186/s13059-017-1381-1 · Cited by 1,497
Adamson, B., Norman, T. M., Jost, M., Cho, M. Y., Nuñez, J. K., Chen, Y., Villalta, J. E., Gilbert, L. A., Horlbeck, M. A., Hein, M. Y., et al. (2016) · Cell · doi:10.1016/j.cell.2016.11.048 · Cited by 1,193 · cited 12×
“Nonetheless, the fine-tuned version of scGPT achieved the highest accuracies overall, outperforming other methods in Adamson 13 , Norman 14 , and Replogle 15 (K562) datasets.”
InfluentialGSEApy: a comprehensive package for performing gene set enrichment analysis in Python
Fang, Z., Liu, X., & Peltz, G. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btac757 · Cited by 1,170
Research works printed but not linked: 2
- Hetzel, L. et al. Predicting cellular responses to novel drug perturbations at a single-cell resolution. Adv. Neural Info. Process. Syst. 35, 26711–26722 (2022).
- Lopez, R., Hütter, J. C., Pritchard, J. & Regev, A. Large-scale differentiable causal discovery of factor graphs. Adv. Neural Info. Process. Syst. 35, 19290–303 (2022).
Cited by
A Systematic Comparison of Single-Cell Perturbation Response Prediction Models
Li, L., You, Y., Fu, Y., Liao, W., Fan, X., Lu, S., Cao, Y., Li, B., Ren, W., Kong, J., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.12.23.630036 · Cited by 25
Deep learning perturbation models can outperform baselines on calibrated metrics
Miller, H. E., Mejia, G. M., Leblanc, F. J. A., Swain, B., Wang, B., & de Lima Camillo, L. P. (2026) · Nature Biotechnology · doi:10.1038/s41587-026-03307-w · Cited by 18
A systematic comparison of single-cell perturbation response prediction models
Li, L., You, Y., Fu, Y., Liao, W., Fan, X., Lu, S., Cao, Y., Li, B., Ren, W., Kong, J., et al. (2026) · Science Advances · doi:10.1126/sciadv.aed3414 · Cited by 1
Evaluating the learnability of single-cell large language models on multiple tasks
Yan, Y., Wang, X., & Song, D. (2026) · doi:10.21203/rs.3.rs-8919408/v1 · Cited by 1
Evaluating the learnability of single-cell large language models on multiple tasks
Yan, Y., Wang, X., & Song, D. (2026) · BMC Genomics · doi:10.1186/s12864-026-12975-6 · Cited by 1
Metadata
- Authors
- Viñas Torné, R., Wiatrak, M., Piran, Z., Fan, S., Jiang, L., Teichmann, S. A., Nitzan, M., & Brbić, M.
- Affiliations
- Wellcome/MRC Cambridge Stem Cell Institute (GB)
- Hebrew University of Jerusalem (IL)
- École Polytechnique Fédérale de Lausanne (CH)
- Canadian Institute for Advanced Research (CA)
- SIB Swiss Institute of Bioinformatics (CH)
- Corresponding author
- Nitzan, Mor
- Brbić, Maria
- Venue
- Nature Biotechnology
- Published
- 2025-08-25
- Volume
- 44
- Issue
- 6
- Pages
- 1050-1059
- ISSN
- 0733-222X (ISSN-L)
- 1087-0156
- 1546-1696
- PMC
- PMC13271886
- OpenAlex
- W4413495491
- Cited by
- 58
- References
- 37
- Citations by year
- 2026: 49 · 2025: 9
- FWCI
- 17.19
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature Biotechnology (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Funders
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (TMSGI2_226252, IC00I0-231922)
- Peter und Traudl Engelhorn Foundation
- European Union
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Chromatin Dynamics
- Keywords
- transcriptional response, systematic variation, selection bias, confounding, evaluation framework, performance indicators, CRISPR/Cas9, cell lines
- MeSH terms
- Humans, Gene Expression Profiling, Genomics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Systema: a framework for evaluating genetic perturbation response prediction beyond systematic variation. https://app.cochat.ai/papers/10.1038/s41587-025-02777-8
- The paper
- Viñas Torné, R., Wiatrak, M., Piran, Z., Fan, S., Jiang, L., Teichmann, S. A., Nitzan, M., & Brbić, M. (2026). Systema: a framework for evaluating genetic perturbation response prediction beyond systematic variation. Nature Biotechnology. https://doi.org/10.1038/s41587-025-02777-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.