CoChat
Sign Up
Cited by 26Gene Regulatory Network AnalysisBioinformatics and Genomic NetworksSingle-cell and spatial transcriptomics

A large-scale benchmark for network inference from single-cell perturbation data

Open access
Chevalley, Mathieu, Roohani, Yusuf H., Mehrjou, Arash, Leskovec, Jure, Schwab, Patrick·2025·Communications Biology
DOI 10.1038/s42003-025-07764-yPMC PMC11897147
Open this paper in CoChat — free account

Abstract

Mapping biological mechanisms in cellular systems is a fundamental step in early-stage drug discovery that serves to generate hypotheses on what disease-relevant molecular targets may effectively be modulated by pharmacological interventions. With the advent of high-throughput methods for measuring single-cell gene expression under genetic perturbations, we now have effective means for generating evidence for causal gene-gene interactions at scale. However, evaluating the performance of network inference methods in real-world environments is challenging due to the lack of ground-truth knowledge. Moreover, traditional evaluations conducted on synthetic datasets do not reflect the performance in real-world systems. We thus introduce CausalBench, a benchmark suite revolutionizing network inference evaluation with real-world, large-scale single-cell perturbation data. CausalBench, distinct from existing benchmarks, offers biologically-motivated metrics and distribution-based interventional measures, providing a more realistic evaluation of network inference methods. An initial systematic evaluation of state-of-the-art causal inference methods using our CausalBench suite highlights how poor scalability of existing methods limits performance. Moreover, methods that use interventional information do not outperform those that only use observational data, contrary to what is observed on synthetic benchmarks. CausalBench subsequently enables the development of numerous promising methods through a community challenge, thus demonstrating its potential as a transformative tool in the field of computational biology, bridging the gap between theoretical innovation and practical application in drug discovery and disease understanding. Thus, CausalBench opens new avenues for method developers in causal network inference research, and provides to practitioners a principled and reliable way to track progress in network methods for real-world interventional data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s42003-025-07764-y

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Despite the rapid growth of perturbation datasets in size and scope, proportional gains in predictive capabilities have not been achieved.

    “Despite the rapid growth of perturbation datasets in size and scope, proportional gains in predictive capabilities have not been achieved.”

    From Predicting cellular responses to perturbation across diverse contexts with State (2026) · cites this paper as [16]

    In Results, under “Characteristics of an optimal method for this task” (page unavailable), the paper reports that most methods gained little from additional perturbations and that nearly all challenge methods plateaued at about 50% of samples. This supports the narrower point that more data does not consistently improve network-inference performance. It does not establish a comparison between the growth of perturbation datasets over time and proportional gains in predictive capabilities; it also reports substantial improvements by newer methods. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 74

  1. [1]Farmer, R. E. et al. Application of causal inference methods in the analyses of randomised controlled trials: a systematic review. Trials 19, 1–14 (2018). 29321046 PMC5761133 · DOI
  2. [2]Baum-Snow, N. & Ferreira, F. Causal inference in urban and regional economics. In Handbook of Regional and Urban Economics, Vol. 5, 3–68 (Elsevier, 2015).
  3. [3]Joffe, M., Gambhir, M., Chadeau-Hyam, M. & Vineis, P. Causal diagrams in systems epidemiology. Emerg. Themes Epidemiol. 9, 1–18 (2012). 22429606 PMC3382427 · DOI
  4. [4]Gentzel, A., Garant, D. & Jensen, D. The case for evaluating causal models using interventional measures and empirical data. Adv. Neural Inf. Process. Syst.32 (2019).
  5. [9]Datlinger, P. et al. Ultra-high-throughput single-cell RNA sequencing and perturbation screening with combinatorial fluidic indexing. Nat. Methods 18, 635–642 (2021). 34059827 PMC7612019 · DOI
  6. [10]Yu, J., Smith, V. A., Wang, P. P., Hartemink, A. J. & Jarvis, E. D. Advances to Bayesian network inference for generating causal networks from observational biological data. Bioinformatics 20, 3594–3603 (2004). 15284094 · DOI · Paper page
  7. [11]Chai, L. E. et al. A review on the computational approaches for gene regulatory network construction. Comput. Biol. Med. 48, 55–65 (2014). 24637147 · DOI
  8. [12]Akers, K. & Murali, T. Gene regulatory network inference in single-cell biology. Curr. Opin. Syst. Biol. 26, 87–97 (2021). · DOI
  9. [13]Hu, X., Hu, Y., Wu, F., Leung, R. W. T. & Qin, J. Integration of single-cell multi-omics for gene regulatory network inference. Comput. Struct. Biotechnol. J. 18, 1925–1938 (2020). 32774787 PMC7385034 · DOI
  10. [14]Neal, B., Huang, C.-W. & Raghupathi, S. Realcause: realistic causal inference benchmarking. arXiv preprint arXiv:2011.15007 (2020). · arXiv
  11. [15]Shimoni, Y., Yanover, C., Karavani, E. & Goldschmnidt, Y. Benchmarking framework for performance-evaluation of causal inference analysis. arXiv preprint arXiv:1802.05046 (2018). · arXiv
  12. [16]Parikh, H., Varjao, C., Xu, L. & Tchetgen, E. Validating causal inference methods. In Proc.International Conference on Machine Learning, 17346–17358 (PMLR, 2022).
  13. [17]Chevalley, M. et al. The CausalBench challenge: A machine learning contest for gene network inference from single-cell perturbation data. arXiv preprint arXiv:2308.15395 (2023). · arXiv
  14. [19]Larson, M. H. et al. Crispr interference (CRISPRi) for sequence-specific control of gene expression. Nat. Protoc. 8, 2180–2196 (2013). 24136345 PMC3922765 · DOI
  15. [20]Spirtes, P., Glymour, C. N., Scheines, R. & Heckerman, D. Causation, Prediction, and Search (MIT Press, 2000).
  16. [22]Zheng, X., Aragam, B., Ravikumar, P. & Xing, E. P. DAGs with NO TEARS: Continuous Optimization for Structure Learning. In Advances in Neural Information Processing Systems (2018).
  17. [23]Zheng, X., Dan, C., Aragam, B., Ravikumar, P. & Xing, E. P. Learning sparse nonparametric DAGs. In Proc.International Conference on Artificial Intelligence and Statistics (2020).
  18. [24]Reisach, A., Seiler, C. & Weichwald, S. Beware of the simulated dag! causal discovery benchmarks may be easy to game. Adv. Neural Inf. Process. Syst. 34, 27772–27784 (2021).
  19. [25]Aibar, S. et al. Scenic: single-cell regulatory network inference and clustering. Nat. Methods 14, 1083–1086 (2017). 28991892 PMC5937676 · DOI · Paper page
  20. [27]Brouillard, P., Lachapelle, S., Lacoste, A., Lacoste-Julien, S. & Drouin, A. Differentiable causal discovery from interventional data. Adv. Neural Inf. Process. Syst. 33, 21865–21877 (2020).
  21. [28]Lopez, R., Hütter, J.-C., Pritchard, J. K. & Regev, A. Large-scale differentiable causal discovery of factor graphs. Adv. Neural Inf. Process. Syst.35, 19290–19303 (2022).
  22. [30]Deng, K. & Guan, Y. A supervised light gum-based approach to the gsk. ai causalbench challenge (ICLR 2023) (2023).
  23. [34]Misiakos, P., Wendler, C. & Püschel, M. Learning dags from data with few root causes. Adv. Neural Inf. Process. Syst.36, 16865–16888 (2023).
  24. [35]Gillis, J., Ballouz, S. & Pavlidis, P. Bias tradeoffs in the creation and analysis of protein–protein interaction networks. J. Proteom. 100, 44–54 (2014). PMC3972268 24480284 · DOI
  25. [36]Carthew, R. W. Gene regulation by micrornas. Curr. Opin. Genet. Dev. 16, 203–208 (2006). 16503132 · DOI
  26. [37]Levine, M. & Davidson, E. H. Gene regulatory networks for development. Proc. Natl Acad. Sci. 102, 4936–4942 (2005). 15788537 PMC555974 · DOI · Paper page
  27. [38]Tracy, S., Yuan, G.-C. & Dries, R. Rescue: imputing dropout events in single-cell rna-sequencing data. BMC Bioinforma. 20, 1–11 (2019). PMC6624880 31299886 · DOI
  28. [39]Jiang, R., Sun, T., Song, D. & Li, J. J. Statistics or biology: the zero-inflation controversy about scrna-seq data. Genome Biol. 23, 31 (2022). 35063006 PMC8783472 · DOI · Paper page
  29. [41]Kowalczyk, M. S. et al. Single-cell RNA-seq reveals changes in cell cycle and differentiation programs upon aging of hematopoietic stem cells. Genome Res. 25, 1860–1872 (2015). 26430063 PMC4665007 · DOI · Paper page
  30. [42]Freedman, D. A. Bootstrapping regression models. Ann. Stat. 9, 1218–1228 (1981). · DOI
  31. [44]Meinshausen, N. & Bühlmann, P. Stability selection. J. R. Stat. Soc. Ser. B: Stat. Methodol. 72, 417–473 (2010). · DOI
  32. [45]Shen, X., Bühlmann, P. & Taeb, A. Causality-oriented robustness: exploiting general additive interventions. arXiv preprint arXiv:2307.10299 (2023). · arXiv
  33. [47]Pearl, J.Causality (Cambridge university press, 2009).
  34. [48]Peters, J., Janzing, D. & Schölkopf, B.Elements of Causal Inference: Foundations and Learning Algorithms (The MIT Press, 2017).
  35. [49]Huang, B., Zhang, K., Lin, Y., Schölkopf, B. & Glymour, C. Generalized score functions for causal discovery. In Proc. 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 1551–1560 (2018). PMC6123020 30191079
  36. [50]Katz, D., Shanmugam, K., Squires, C. & Uhler, C. Size of interventional Markov equivalence classes in random dag models. In Proc. 22nd International Conference on Artificial Intelligence and Statistics, 3234–3243 (PMLR, 2019).
  37. [51]Mehrjou, A. et al. GeneDisco: a benchmark for experimental design in drug discovery. In Proc.International Conference on Learning Representations (2022).
  38. [53]Wang, Y., Solus, L., Yang, K. & Uhler, C. Permutation-based causal inference algorithms with interventions. Adv. Neural Inf. Process. Syst.30 (2017).
  39. [54]Schölkopf, B. et al. Toward causal representation learning. Proc. IEEE 109, 612–634 (2021). · DOI · Paper page
  40. [55]Ke, N. R. et al. Learning neural causal models from unknown interventions. arXiv preprint arXiv:1910.01075 (2019). · arXiv
  41. [56]Scherrer, N. et al. Learning neural causal models with active interventions. arXiv preprint arXiv:2109.02429 (2021). · arXiv
  42. [57]Friedman, N., Linial, M., Nachman, I. & Pe’er, D. Using Bayesian networks to analyze expression data. J. Comput. Biol. 7, 601–620 (2000). 11108481 · DOI
  43. [58]Kamimoto, K., Hoffmann, C. M. & Morris, S. A. Dissecting cell identity via network inference and in silico gene perturbation. Nature614, 742–751 (2023). PMC9946838 36755098
  44. [62]Li, Y. et al. Enhancer-driven gene regulatory networks inference from single-cell RNA-seq and ATAC-seq data. Brief. Bioinforma. 25, bbae369 (2024). PMC11289686 39082647 · DOI
  45. [63]Cao, Z.-J. & Gao, G. Multi-omics single-cell data integration and regulatory inference with graph-linked embedding. Nat. Biotechnol. 40, 1458–1466 (2022). 35501393 PMC9546775 · DOI · Paper page
  46. [64]Zhang, L., Zhang, J. & Nie, Q. Direct-net: An efficient method to discover cis-regulatory elements and construct regulatory networks from single-cell multiomics data. Sci. Adv. 8, eabl7393 (2022). 35648859 PMC9159696 · DOI
  47. [65]Fleck, J. S. et al. Inferring and perturbing cell fate regulomes in human brain organoids. Nature 621, 365–372 (2023). 36198796 PMC10499607 · DOI · Paper page
  48. [66]Li, Z., Nagai, J. S., Kuppe, C., Kramann, R. & Costa, I. G. scmega: single-cell multi-omic enhancer-based gene regulatory network inference. Bioinforma. Adv. 3, vbad003 (2023). PMC9853317 36698768 · DOI
  49. [68]Barabási, A.-L. & Albert, R. Emergence of scaling in random networks. Science 286, 509–512 (1999). 10521342 · DOI · Paper page
  50. [69]Peters, J. & Bühlmann, P. Structural intervention distance for evaluating causal graphs. Neural Comput. 27, 771–799 (2015). 25602767 · DOI
  51. [70]Dibaeinia, P. & Sinha, S. Sergio: a single-cell expression simulator guided by gene regulatory networks. Cell Syst. 11, 252–271 (2020). 32871105 PMC7530147 · DOI
  52. [71]Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011). 22006061 · DOI · Paper page
  53. [72]Marbach, D. et al. Wisdom of crowds for robust gene network inference. Nat. Methods 9, 796–804 (2012). 22796662 PMC3512113 · DOI · Paper page
  54. [73]Peidli, S. et al. scperturb: harmonized single-cell perturbation data. Nat. Methods21, 531–540 (2024). PMC12220817 38279009
  55. [74]Giurgiu, M. et al. Corum: the comprehensive resource of mammalian protein complexes—2019. Nucleic acids Res. 47, D559–D563 (2019). 30357367 PMC6323970 · DOI · Paper page
  56. [75]Von Mering, C. et al. String: known and predicted protein–protein associations, integrated and transferred across organisms. Nucleic Acids Res. 33, D433–D437 (2005). 15608232 PMC539959 · DOI
  57. [76]Snel, B., Lehmann, G., Bork, P. & Huynen, M. A. String: a web-server to retrieve and display the repeatedly occurring neighbourhood of a gene. Nucleic Acids Res. 28, 3442–3444 (2000). 10982861 PMC110752 · DOI
  58. [77]Von Mering, C. et al. String 7—recent developments in the integration and prediction of protein interactions. Nucleic Acids Res. 35, D358–D362 (2007). 17098935 PMC1669762 · DOI
  59. [78]Jensen, L. J. et al. String 8—a global view on proteins and their functional interactions in 630 organisms. Nucleic Acids Res. 37, D412–D416 (2009). 18940858 PMC2686466 · DOI
  60. [79]Mering, C. V. et al. String: a database of predicted functional associations between proteins. Nucleic Acids Res. 31, 258–261 (2003). 12519996 PMC165481 · DOI
  61. [80]Szklarczyk, D. et al. The string database in 2011: functional interaction networks of proteins, globally integrated and scored. Nucleic Acids Res. 39, D561–D568 (2010). 21045058 PMC3013807 · DOI
  62. [81]Szklarczyk, D. et al. String v10: protein–protein interaction networks, integrated over the tree of life. Nucleic Acids Res. 43, D447–D452 (2015). 25352553 PMC4383874 · DOI · Paper page
  63. [82]Szklarczyk, D. et al. The string database in 2017: quality-controlled protein–protein association networks, made broadly accessible. Nucleic Acids Research gkw937 (2016). PMC5210637 27924014
  64. [83]Szklarczyk, D. et al. String v11: protein–protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets. Nucleic Acids Res. 47, D607–D613 (2019). 30476243 PMC6323986 · DOI · Paper page
  65. [84]Szklarczyk, D. et al. The string database in 2021: customizable protein–protein networks, and functional characterization of user-uploaded gene/measurement sets. Nucleic Acids Res. 49, D605–D612 (2021). 33237311 PMC7779004 · DOI · Paper page
  66. [85]Franceschini, A. et al. String v9. 1: protein-protein interaction networks, with increased coverage and integration. Nucleic Acids Res. 41, D808–D815 (2012). 23203871 PMC3531103 · DOI · Paper page
  67. [86]Franceschini, A., Lin, J., von Mering, C. & Jensen, L. J. Svd-phy: improved prediction of protein functional associations through singular value decomposition of phylogenetic profiles. Bioinformatics 32, 1085–1087 (2016). 26614125 PMC4896368 · DOI
  68. [88]Bucchianico, A. D. Combinatorics, computer algebra, and the Wilcoxon-Mann-Whitney test”. J. Stat. Plan. Inference 79, 349–364 (1999). · DOI
  69. [89]Zou, Z., Ohta, T., Miura, F. & Oki, S. Chip-atlas 2021 update: a data-mining suite for exploring epigenomic landscapes by fully integrating chip-seq, ATAC-seq and bisulfite-seq data. Nucleic Acids Res.50, W175–W182 (2022). PMC9252733 35325188
  70. [90]Davis, C. A. et al. The encyclopedia of DNA elements (encode): data portal update. Nucleic Acids Res. 46, D794–D801 (2018). 29126249 PMC5753278 · DOI · Paper page
  71. [92]Pickrell, J. K., Gaffney, D. J., Gilad, Y. & Pritchard, J. K. False positive peaks in chip-seq and other sequencing-based functional assays caused by unannotated high copy number regions. Bioinformatics 27, 2144–2146 (2011). 21690102 PMC3137225 · DOI
  72. [93]Ramdas, A., García Trillos, N. & Cuturi, M. On Wasserstein two-sample testing and related families of nonparametric tests. Entropy 19, 47 (2017). · DOI
  73. [95]Virtanen, P. et al. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat. Methods 17, 261–272 (2020). 32015543 PMC7056644 · DOI · Paper page
  74. [96]Lorch, L., Sussex, S., Rothfuss, J., Krause, A. & Schölkopf, B. Amortized inference for causal structure learning. Adv. Neural Inf. Process. Syst.35, 13104–13118 (2022).

Cited by

Sorted by citations

Authors
Chevalley, M., Roohani, Y. H., Mehrjou, A., Leskovec, J., & Schwab, P.
Affiliations
  • ETH Zurich (CH)
  • GlaxoSmithKline (Switzerland) (CH)
  • Stanford University (US)
Corresponding author
Schwab, Patrick
ORCID
Venue
Communications Biology
Publisher
Springer Science and Business Media LLC
Published
2025-03-11
Volume
8
Issue
1
Pages
412-412
ISSN
2399-3642 (ISSN-L)
DOI
10.1038/s42003-025-07764-y
PMC
PMC11897147
OpenAlex
W4408327225
Cited by
26
References
12
Citations by year
2026: 21 · 2025: 5
FWCI
7.82
Type
journal article
Language
en
Open access
Licence
Locations
  • Communications Biology (publishedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Repository for Publications and Research Data (ETH Zurich) (publishedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • ETH Zürich Research Collection
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
Keywords
causal network inference, CausalBench, benchmark suite, gene-gene interaction, interventional data, scalability, drug discovery, single-cell gene expression, community problems
MeSH terms
Algorithms, Humans, Computational Biology, Benchmarking, Gene Regulatory Networks, Single-Cell Analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review