Towards a Rigorous Assessment of Systems Biology Models: The DREAM3 Challenges
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Systems biology has embraced computational modeling in response to the quantitative nature and increasing scale of contemporary data sets. The onslaught of data is accelerating as molecular profiling technology evolves. The Dialogue for Reverse Engineering Assessments and Methods (DREAM) is a community effort to catalyze discussion about the design, application, and assessment of systems biology models through annual reverse-engineering challenges. METHODOLOGY AND PRINCIPAL FINDINGS: We describe our assessments of the four challenges associated with the third DREAM conference which came to be known as the DREAM3 challenges: signaling cascade identification, signaling response prediction, gene expression prediction, and the DREAM3 in silico network challenge. The challenges, based on anonymized data sets, tested participants in network inference and prediction of measurements. Forty teams submitted 413 predicted networks and measurement test sets. Overall, a handful of best-performer teams were identified, while a majority of teams made predictions that were equivalent to random. Counterintuitively, combining the predictions of multiple teams (including the weaker teams) can in some cases improve predictive power beyond that of any single method. CONCLUSIONS: DREAM provides valuable feedback to practitioners of systems biology modeling. Lessons learned from the predictions of the community provide much-needed context for interpreting claims of efficacy of algorithms described in the scientific literature.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Their results show that, in addition to the obstacles identified by Prill et al. [94] (lack of data and deficiencies in the inference algorithms), we must be also aware of fundamental problems related to the uniqueness and distinguishability of these networks (even for the utopian case of fully observed networks with no noise).”
From Reverse engineering and identification in systems biology: strategies, perspectives and challenges (2014) · cites this paper as [94]
In the citing paper
“The organizers identify two major hurdles to be surmounted: lack of data and deficiencies in the inference algorithms.”
From Reverse engineering and identification in systems biology: strategies, perspectives and challenges (2014) · cites this paper as [94]
In the citing paper
“In other words, reliable network inference remains an unsolved problem.”
From Reverse engineering and identification in systems biology: strategies, perspectives and challenges (2014) · cites this paper as [94]
In the citing paper
“There are several notable examples where collaborations have been very successful, such as SBML [238], BioModels Database [239] or the DREAM challenges [94].”
From Reverse engineering and identification in systems biology: strategies, perspectives and challenges (2014) · cites this paper as [94]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Exploration, normalization, and summaries of high density oligonucleotide array probe level data
Irizarry, R. A. (2003) · Biostatistics · doi:10.1093/biostatistics/4.2.249 · Cited by 10,603 · cited 1×
“Measurements were normalized using the RMA algorithm [18] within the commercial software package, GeneSpring.”
MethodJournal of the Royal Statistical Society (B)
Fildes, R. (1993) · International Journal of Forecasting · doi:10.1016/0169-2070(93)90088-5 · Cited by 3,437
Margolin, A. A., Nemenman, I., Basso, K., Wiggins, C., Stolovitzky, G., Favera, R. D., & Califano, A. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-s1-s7 · Cited by 2,420 · cited 1×
“…be applied off-the-shelf, a strategy for guarding against bogus predictions is to employ a wide variety of algorithms that train on different features of the data (e.g., correlation [28], mutual information [29–32], synergy [33], etc.), then merge the results to gain confidence in the predictions.”
ResultSelf-citationMargolin, A. A., Nemenman, I. M., Basso, K., Wiggins, C. H., Stolovitzky, G. A., Favera, R. D., & Califano, A. (2006) · BMC Bioinformatics · doi:10.7916/d8g15zbp · Cited by 2,246
Self-citationReverse engineering of regulatory networks in human B cells
Basso, K., Margolin, A. A., Stolovitzky, G., Klein, U., Dalla-Favera, R., & Califano, A. (2005) · Nature Genetics · doi:10.1038/ng1532 · Cited by 1,404 · cited 1×
“, correlation [28], mutual information [29–32], synergy [33], etc.”
BackgroundSelf-citationBUTTE, A. J., & KOHANE, I. S. (1999) · Biocomputing 2000 · doi:10.1142/9789814447331_0040 · Cited by 945 · cited 1×
“, correlation [28], mutual information [29–32], synergy [33], etc.”
BackgroundNatarajan, K., Meyer, M. R., Jackson, B. M., Slade, D., Roberts, C., Hinnebusch, A. G., & Marton, M. J. (2001) · Molecular and Cellular Biology · doi:10.1128/mcb.21.13.4347-4368.2001 · Cited by 748 · cited 1×
“Expression levels were assayed separately in the three mutant strains and in the wild type strain at times 0, 10, 20, 30, 45, 60, 90 and 120 minutes following the addition of 3-aminotriazole (3AT) as described [17].”
MethodButte, A. J., Tamayo, P., Slonim, D., Golub, T. R., & Kohane, I. S. (2000) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.220392197 · Cited by 605 · cited 1×
“…be applied off-the-shelf, a strategy for guarding against bogus predictions is to employ a wide variety of algorithms that train on different features of the data (e.g., correlation [28], mutual information [29–32], synergy [33], etc.), then merge the results to gain confidence in the predictions.”
ResultBonneau, R., Reiss, D. J., Shannon, P., Facciotti, M., Hood, L., Baliga, N. S., & Thorsson, V. (2006) · Genome Biology · doi:10.1186/gb-2006-7-5-r36 · Cited by 510
Research works printed but not linked: 1
- [15]Alexopoulos L Saez-Rodriguez J Cosgrove B Lauffenburger D PK S 2009 Networks reconstructed from cell response data reveal profound dif ferences in signaling by tolllike receptors and nf-?b in normal and transformed human hepatocytes. Under review
Cited by
Items 101–200 of 427
Petri Nets with Fuzzy Logic (PNFL): Reverse Engineering and Parametrization
Küffner, R., Petri, T., Windhager, L., & Zimmer, R. (2010) · PLoS ONE · doi:10.1371/journal.pone.0012807 · Cited by 51
“The evaluation is described in more detail in [19].”
Time lagged information theoretic approaches to the reverse engineering of gene regulatory networks
Chaitankar, V., Ghosh, P., Perkins, E. J., Gong, P., & Zhang, C. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-s6-s19 · Cited by 45
“We generated a 20 gene network with 24 time point’s data using the Genenetweaver tool [21-23].”
MethodSimultaneous model discrimination and parameter estimation in dynamic models of cellular systems
Rodriguez-Fernández, M., Rehberg, M., Kremling, A., & Banga, J. R. (2013) · BMC Systems Biology · doi:10.1186/1752-0509-7-76 · Cited by 50
“…plus the parameters from a set of data), we consider a subproblem which is smaller (although still very challenging) and dense (so sparsity is not an issue), and which, therefore, does not suffer from many of the ill-posedness and ill-conditioning maladies of the general inference problem [51,52].”
BackgroundGoswami, C. P., Cheng, L., Alexander, P., Singal, A. G., & Li, L. (2015) · CPT Pharmacometrics & Systems Pharmacology · doi:10.1002/psp4.9 · Cited by 46
Network inference using asynchronously updated kinetic Ising model
Zeng, H.-L., Aurell, E., Alava, M. J., & Mahmoudi, H. (2011) · Physical Review E · doi:10.1103/physreve.83.041135 · Cited by 49
Discriminating direct and indirect connectivities in biological networks
Kang, T., Moore, R. L., Li, Y., Sontag, E. D., & Bleris, L. G. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1507168112 · Cited by 42
“The majority of efforts to address the verification issue adopt in silico benchmark suites that are based on biological pathway approximations (19).”
BackgroundTime-Based Systems Biology Approaches to Capture and Model Dynamic Gene Regulatory Networks
Álvarez, J. M., Brooks, M. D., Swift, J., & Coruzzi, G. M. (2021) · Annual Review of Plant Biology · doi:10.1146/annurev-arplant-081320-090914 · Cited by 44
Grzegorczyk, M. A., & Husmeier, D. (2013) · Machine Learning · doi:10.1007/s10994-012-5326-3 · Cited by 36
Liang, Y., & Kelemen, A. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx036 · Cited by 41
“boosting and bagging) which combine the predictions of a wide variety of algorithms on different aspects and features of the data into a consensus/aggregate prediction for guarding against bogus predictions, then merge the results to gain confidence to learn new biology from the aggregate predictions of the systems bi…”
BackgroundResultInfluentialShojaie, A., Jauhiainen, A., Kallitsis, M., & Michailidis, G. C. (2014) · PLoS ONE · doi:10.1371/journal.pone.0082393 · Cited by 40
“To illustrate the performance of the RIPE algorithm, we first evaluate it by using networks from the DREAM4 in-silico network challenge, which is part of a series of challenges [32], aiming at inferring gene regulatory networks from simulated data.”
MethodComputational dynamic approaches for temporal omics data with applications to systems medicine
Liang, Y., & Kelemen, A. (2017) · BioData Mining · doi:10.1186/s13040-017-0140-x · Cited by 38
“…provided excellent examples for temporal omics data sets that involve various most updated biomedical challenge questions (e.g. regulatory network inference, causal inferences, dynamic trajectory predictions) through multiple team competitions [16, 17, 86, 118, 119, 131, 132].”
BackgroundGrzegorczyk, M. A., & Husmeier, D. (2012) · Statistical Applications in Genetics and Molecular Biology · doi:10.1515/1544-6115.1761 · Cited by 36
Semi-supervised prediction of gene regulatory networks using machine learning algorithms
Patel, N., & Wang, J. T. L. (2015) · Journal of Biosciences · doi:10.1007/s12038-015-9558-9 · Cited by 39
“For S. cerevisiae, there were three regulatory databases used by the DREAM5 study (Marbach et al. 2009, 2010, 2012; Prill et al. 2010 et al. 2006), and the database described in (Hu et al. 2007).”
MethodGustafsson, M., & Hörnquist, M. (2010) · PLoS ONE · doi:10.1371/journal.pone.0009134 · Cited by 35
miRLAB: An R Based Dry Lab for Exploring miRNA-mRNA Regulatory Relationships
Le, T. D., Zhang, J., Liu, L., Liu, H., & Li, J. (2015) · PLoS ONE · doi:10.1371/journal.pone.0145386 · Cited by 38
“Z-score [34] is a network inference method to estimate the effects of gene knockout experiment.”
MethodReconstructing directional causal networks with random forest: Causality meeting machine learning
Leng, S., Xu, Z., & Ma, H. (2019) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.5120778 · Cited by 36
Faria, J. P., Overbeek, R., Xia, F., Rocha, M., Rocha, I., & Henry, C. S. (2013) · Briefings in Bioinformatics · doi:10.1093/bib/bbs071 · Cited by 41
Gupta, R., Stincone, A., Antczak, P., Durant, S., Bicknell, R., Bikfalvi, A., & Falciani, F. (2011) · BMC Systems Biology · doi:10.1186/1752-0509-5-52 · Cited by 38
“This network generator has been used as part of the DREAM Initiative [20].”
MethodLim, N., Şenbabaoğlu, Y., Michailidis, G. C., & d’Alché–Buc, F. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt167 · Cited by 34
“Although there is no information on the structure of team 236’s algorithm, its authors responded to the post-competition DREAM3 survey stating that their method employs Bayesian models with an in-degree constraint (Prill et al., 2010).”
BackgroundMethodRigorous assessment of gene set enrichment tests
Naeem, H., Zimmer, R., Tavakkolkhah, P., & Küffner, R. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts164 · Cited by 34
A Crowdsourcing Approach to Developing and Assessing Prediction Algorithms for AML Prognosis
Noren, D. P., Long, B. L., Norel, R., Rrhissorrakrai, K., Hess, K. R., Hu, C. W., Bisberg, A. J., Schultz, A., Engquist, E., Liu, L., et al. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004890 · Cited by 32
Self-citationWhat do ’omic technologies have to offer periodontal clinical practice in the future?
Grant, M. M. (2011) · Journal of Periodontal Research · doi:10.1111/j.1600-0765.2011.01387.x · Cited by 36
Inferring Phenotypic Properties from Single-Cell Characteristics
Qiu, P. (2012) · PLoS ONE · doi:10.1371/journal.pone.0037038 · Cited by 31
“The challenges put forth by DREAM, the acronym for Dialogue for Reverse Engineering Assessments and Methods, provide objective and unbiased platforms for evaluating computational methods in systems biology [24–27].”
Jiang, J., Chen, S., Tsou, T., McGinnis, C. S., Khazaei, T., Zhu, Q., Park, J. H., Strazhnik, I.-M., Vielmetter, J., Gong, Y., et al. (2026) · Cell · doi:10.1016/j.cell.2026.04.028 · Cited by 37
Ensemble Methods for MiRNA Target Prediction from Expression Data
Le, T. D., Zhang, J., Liu, L., & Li, J. (2015) · PLoS ONE · doi:10.1371/journal.pone.0131627 · Cited by 40
“Z-score [53] is a network inference method to estimate the effects of gene knockout experiment.”
MethodJoint estimation of causal effects from observational and intervention gene expression data
Rau, A., Jaffrézic, F., & Nuel, G. (2013) · BMC Systems Biology · doi:10.1186/1752-0509-7-111 · Cited by 32
“DREAM data analysis The proposed MCMC-Mallows algorithm as well as the two previously presented methods [7,11] were applied to data from the DREAM4 challenge, an international competition held yearly to contribute to the development of powerful inference methods [13-16].”
MethodInformation theoretic approaches for inference of biological networks from continuous-valued data
Budden, D., & Crampin, E. J. (2016) · BMC Systems Biology · doi:10.1186/s12918-016-0331-y · Cited by 33
“Early DREAM challenges focused primarily on simulated data-sets, whereby a ‘true’ network topology was used to generate artificial gene expression data [6, 7].”
BackgroundAn Information-Theoretic Machine Learning Approach to Expression QTL Analysis
Huang, T., & Cai, Y. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067899 · Cited by 29
Xiong, J., & Zhou, T. (2012) · PLoS ONE · doi:10.1371/journal.pone.0043819 · Cited by 29
“More detailed explanations can be found in [22] or on the web site of the DREAM project at http://wiki.c2b2.columbia.edu/ dream/.”
BackgroundMethodInfluentialTitsias, M. K., Honkela, A., Lawrence, N. D., & Rattray, M. (2012) · BMC Systems Biology · doi:10.1186/1752-0509-6-53 · Cited by 34
“The Inferelator was one of the top performing methods for GRN inference in recent Dialogue for Reverse Engineering Assessments and Methods (DREAM) competitions for network inference [23] (DREAM 3 [24] and DREAM 4 [25]).”
MethodA modular framework for gene set analysis integrating multilevel omics data
Sass, S., Buettner, F., Mueller, N. S., & Theis, F. J. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt752 · Cited by 31
“The development of more and more powerful techniques for the inference of gene interactions (39) leads to a comprehensive and reliable knowledge of gene regulation and may improve the outcome of the MONA algorithm.”
A novel quantum inspired algorithm for sparse fuzzy cognitive maps learning
Kolahdoozi, M., Amirkhani, A., Shojaeefard, . H., & Abraham, A. (2019) · Applied Intelligence · doi:10.1007/s10489-019-01476-7 · Cited by 30
Reverse engineering gene regulatory networks from measurement with missing values
Ogundijo, O., Elmas, A., & Wang, X. (2016) · EURASIP Journal on Bioinformatics and Systems Biology · doi:10.1186/s13637-016-0055-8 · Cited by 28
An Integrated Approach to Reconstructing Genome-Scale Transcriptional Regulatory Networks
Imam, S., Noguera, D. R., & Donohue, T. J. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004103 · Cited by 28
“standard of true positives interactions (TP) [43].”
MethodAddressing Information Asymmetry: Deep Temporal Causality Discovery for Mixed Time Series
Chen, J., & Zhao, C. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3553957 · Cited by 36
Stability Indicators in Network Reconstruction
Filosi, M., Visintainer, R., Riccadonna, S., Jurman, G., & Furlanello, C. (2014) · PLoS ONE · doi:10.1371/journal.pone.0089815 · Cited by 30
“…accuracy is far from being optimal in many situations with the presence of several pitfalls (Meyer et al., 2011), related to both the methods and the data (He et al., 2009), with the extreme situation of many link prediction being statistically equivalent to random guesses (Prill et al., 2010).”
BackgroundBiological Network Inference: A Review of Methods and Assessment of Tools and Techniques
Omony, J. (2014) · Annual Research & Review in Biology · doi:10.9734/arrb/2014/5718 · Cited by 31
“DREAM1 [36], DREAM2 [37,38] and DREAM3 [35,39].”
LiPLike: towards gene regulatory network predictions of high certainty
Magnusson, R., & Gustafsson, M. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz950 · Cited by 26
“The overwhelming size and complexity of data from modern techniques, such as RNA-Seq hinder the drawing of conclusions from straightforward analyses (Magnusson et al. , 2017).”
A probabilistic graphical model for system-wide analysis of gene regulatory networks
Kotiang, S., & Eslami, A. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa122 · Cited by 33
Improving gene regulatory network inference using network topology information
Nair, A., Chetty, M., & Wangikar, P. P. (2015) · Molecular BioSystems · doi:10.1039/c5mb00122f · Cited by 27
Can biological complexity be reverse engineered?
Green, S. (2015) · Studies in History and Philosophy of Science Part C Studies in History and Philosophy of Biological and Biomedical Sciences · doi:10.1016/j.shpsc.2015.03.008 · Cited by 29
“An example is the DREAM project, Dialogue for Reverse-Engineering Assessment and Methods (Stolovitzky et al., 2007; Prill et al., 2010).”
MethodApproximate inference of gene regulatory network models from RNA-Seq time series data
Thorne, T. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2125-2 · Cited by 35
“Subnetworks of 50 nodes were generated and used to simulate 21 time points with 10 replicates, as in the DREAM4 challenge[39, 31, 32].”
MethodEconomy Statistical Recurrent Units For Inferring Nonlinear Granger CausalityPreprint
Khanna, S., & Tan, V. Y. F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09879 · Cited by 68
“In the final set of experiments, we evaluate the performance of the different time series models in inferring gene regulation networks synthesized for the DREAM-3 In Silico Network Challenge (Prill et al. (2010); Marbach et al. (2009)).”
MethodA Kalman-Filter Based Approach to Identification of Time-Varying Gene Regulatory Networks
Xiong, J., & Zhou, T. (2013) · PLoS ONE · doi:10.1371/journal.pone.0074571 · Cited by 25
“In the rest of this subsection, we will give another simulation, in which gene expression data are from the DREAM3 in silico size10 challenge [23,24].”
Chan, S. C., Zhang, L., Wu, H. C., & Tsui, K. M. (2014) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2014.2343951 · Cited by 23
“mented in the literatures on their functions at the cell cycle TABLE 4 Comparison of Run Time, PPV and Se among Different Algorithms by Applying to a DREAM Data Set [39], [40], [41]”
MethodReconstruction of gene networks using prior knowledge
Ghanbari, M., Lasserre, J., & Vingron, M. (2015) · BMC Systems Biology · doi:10.1186/s12918-015-0233-4 · Cited by 27
“• A synthetic dataset from the DREAM4 competition [20–22].”
MethodA unified framework for estimating parameters of kinetic biological models
Baker, S. M., Poskar, C. H., Schreiber, F., & Junker, B. H. (2015) · BMC Bioinformatics · doi:10.1186/s12859-015-0500-9 · Cited by 24
“MATLAB and used to estimate parameters of two insilico models, a kinetic model for sucrose accumulation in the sugar cane culm tissue [40,41] (SBML model available from the Biomodels database [42]), and a gene regulatory network supplied by the DREAM6 Estimation of Model Parameters Challenge [43] (the SBML model is av…”
MethodGodsey, B. (2013) · PLoS ONE · doi:10.1371/journal.pone.0068358 · Cited by 23
Inference of gene regulatory networks based on directed graph convolutional networks
Wei, P.-J., Guo, Z., Gao, Z., Ding, Z., Cao, R., Su, Y., & Zheng, C.-H. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae309 · Cited by 26
“It includes DREAM4 [31–33] and DREAM5 [34] challenges.”
Penfold, C. A., Shifaz, A., Brown, P. E., Nicholson, A. E., & Wild, D. L. (2015) · Statistical Applications in Genetics and Molecular Biology · doi:10.1515/sagmb-2014-0082 · Cited by 27
Morshed, N., Chetty, M., & Vinh, N. X. (2012) · BMC Systems Biology · doi:10.1186/1752-0509-6-62 · Cited by 22
“This tool has been used to generate in silico benchmarks in the DREAM (both DREAM3 [10] and DREAM4 [11]) challenge initiative.”
Reconstruction of Large-Scale Gene Regulatory Networks Using Bayesian Model Averaging
Kim, H., & Gelenbe, E. (2012) · IEEE Transactions on NanoBioscience · doi:10.1109/tnb.2012.2214233 · Cited by 22
“In order to evaluate the algorithms with non-linear expression data, DREAM4 dataset [12], [36] consisting of 100 nodes was taken and stochastic differential equation approach of GeneNetWeaver [37] was used for the simulation data generation.”
MethodRoth, W., Hecker, D., & Fava, E. (2015) · Methods in molecular biology · doi:10.1007/978-1-4939-2627-5_21 · Cited by 22
Consumer motives and impact of western media on the Moroccan luxury buyer
Hamelin, N., & Thaichon, P. (2016) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2016.06.010 · Cited by 23
PoLoBag: Polynomial Lasso Bagging for signed gene regulatory network inference from expression data
Roy, G. G., Geard, N. L., Verspoor, K. M., & He, S. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa651 · Cited by 27
Jiang, J., Chen, S., Tsou, T., McGinnis, C. S., Khazaei, T., Zhu, Q., Park, J. H., Strazhnik, I.-M., Vielmetter, J., Gong, Y., et al. (2023) · doi:10.1101/2023.04.19.537364 · Cited by 21
Villeneuve, D. L., García‐Reyero, N., Martinović‐Weigelt, D., Li, Z., Watanabe, K. H., Orlando, E. F., LaLone, C. A., Edwards, S. W., Burgoon, L. D., Denslow, N. D., et al. (2011) · Mutation Research/Genetic Toxicology and Environmental Mutagenesis · doi:10.1016/j.mrgentox.2011.12.016 · Cited by 22
Feiglin, A., Hacohen, A., Sarusi-Portuguez, A., Fisher, J., Unger, R., & Ofran, Y. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts517 · Cited by 22
“To determine to what extent complex biological processes can be modeled and analyzed using only the static structure of the network (i.e. the direction and sign of the edges), we attempt to predict the phenotypic effect of perturbations in biological networks from the static network structure.”
BackgroundFisher, C. K., & Mehta, P. (2015) · Bioinform. · doi:10.1093/bioinformatics/btv037 · Cited by 18
“In 2010, the Dialogue for Reverse Engineering Assessments and Methods (DREAM) (Prill et al., 2010) initiative issued a challenge to predict the response of soybean plants to a pathogen from data on gene expression (Zhou et al., 2009).”
BackgroundTechnologies and Approaches to Elucidate and Model the Virulence Program of Salmonella
McDermott, J., Yoon, H., Nakayasu, E. S., Metz, T., Hyduke, D. R., Kidwai, A. S., Palsson, B. Ø., Adkins, J., & Heffron, F. (2011) · Frontiers in Microbiology · doi:10.3389/fmicb.2011.00121 · Cited by 22
“…relies on the similarity between gene profiles as well as a network inference method that assesses the impact of regulator deletions on the expression levels of other regulators to derive a network of regulatory interactions (Yoon et al., 2009; Prill et al., 2010) that was validated using RT-PCR.”
Gene Regulatory Network Review
Liu, E., Li, L., & Cheng, L. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-809633-8.20218-5 · Cited by 25
Causality-Aware Spatiotemporal Graph Neural Networks for Spatiotemporal Time Series Imputation
Jing, B., Zhou, D., Ren, K., & Yang, C. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679642 · Cited by 33
Hub-Centered Gene Network Reconstruction Using Automatic Relevance Determination
Böck, M., Ogishima, S., Tanaka, H., Krämer, S., & Kaderali, L. (2012) · PLoS ONE · doi:10.1371/journal.pone.0035077 · Cited by 23
“This may be due to low quality of the experimental data and the lack of targeted interventions, but these results are in line with findings in recent results of the DREAM competition, where also even the best submitted methods showed surprisingly weak performance, and most methods did not perform better than guessing…”
ResultCausal relationship inference for a large-scale cellular network
Zhou, T., & Wang, Y. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq325 · Cited by 20
Integrated Inference and Analysis of Regulatory Networks from Multi-Level Measurements
Poultney, C. S., Greenfield, A., & Bonneau, R. (2012) · Methods in cell biology · doi:10.1016/b978-0-12-388403-9.00002-3 · Cited by 22
“However, these types of data cover only the transcriptional level of regulation, and even state-of-the-art methods (Greenfield et al., 2010; Huynh-Thu et al., 2010; Pinna et al., 2010; Prill et al., 2010) still make a significant number of false predictions.”
BackgroundMethodInfluentialA crowd-sourcing approach for the construction of species-specific cell signaling networks
Bilal, E., Sakellaropoulos, T., Participants, C., Melas, I. N., Messinis, D. E., Belcastro, V., Rhrissorrakrai, K., Meyer, P., Norel, R., Iskandar, A. R., et al. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu659 · Cited by 21
Self-citationMiller, M. A., Feng, X., Li, G., & RABITZ, H. A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0037664 · Cited by 22
“Amelia II was downloaded as an R-package and implemented as previously described [37,38,60].”
BackgroundMethodInfluentialPredicting miRNA Targets by Integrating Gene Regulatory Knowledge with Expression Profiles
Zhang, W., Le, T. D., Liu, L., Zhou, Z., & Li, J. (2016) · PLoS ONE · doi:10.1371/journal.pone.0152860 · Cited by 20
“We use F-Score to compare CIDER with a variety of popular miRNA target prediction methods, including Pearson correlation [42], Lasso [48], Z-Score [49].”
MethodInferring Gene Regulatory Networks Using Conditional Regulation Pattern to Guide Candidate Genes
Xiao, F., Gao, L., Ye, Y., Hu, Y., & He, R. (2016) · PLoS ONE · doi:10.1371/journal.pone.0154953 · Cited by 23
“We apply our algorithm to DREAM data [2, 26, 27], and experimental results show that RPNI outperforms PCA-CMI and CMI2NI in both accuracy and time complexity.”
MethodFast and accurate gene regulatory network inference by normalized least squares regression
Hillerton, T., Seçilmiş, D., Nelander, S., & Sonnhammer, E. L. L. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac103 · Cited by 20
Multi-omic network inference from time-series data
Garcia, M. M., Aalto, A., Montanari, A. N., Skupin, A., & Gonçalves, J. M. (2025) · npj Systems Biology and Applications · doi:10.1038/s41540-025-00591-1 · Cited by 22
“We compared MINIE with BINGO 3 and dynGENIE3 20 , which are the top-performing methods in the DREAM4 in silico GRN inference challenge 3,45 – 47 .”
Guo, W., Calixto, C. P. G., Τzioutziou, N. A., Lin, P., Waugh, R., Brown, J. W. S., & Zhang, R. (2017) · BMC Systems Biology · doi:10.1186/s12918-017-0440-2 · Cited by 20
“The DREAM3 challenge presents gene network inference problems based on in silico networks of sizes ranging from 10, 50 and 100 genes [20 – 24].”
BackgroundRulNet: A Web-Oriented Platform for Regulatory Network Inference, Application to Wheat –Omics Data
Vincent, J., Martre, P., Gouriou, B., Ravel, C., Dai, Z., Petit, J.-M., & Pailloux, M. (2015) · PLoS ONE · doi:10.1371/journal.pone.0127127 · Cited by 24
“Two standard quality metrics frommachine learning were calculated, the area under the precision-recall (AUPR) and receiver operating characteristic (AUROC) curves [54].”
Reverse engineering gene regulatory networks by modular response analysis – a benchmark
Klinger, B., & Blüthgen, N. (2018) · Essays in Biochemistry · doi:10.1042/ebc20180012 · Cited by 21
Yang, L., Lin, W. J., & Leng, S. (2023) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/5.0144310 · Cited by 18
DTW-MIC Coexpression Networks from Time-Course Data
Riccadonna, S., Jurman, G., Visintainer, R., Filosi, M., & Furlanello, C. (2016) · PLoS ONE · doi:10.1371/journal.pone.0152648 · Cited by 17
“is known. In detail, we demonstrate DTWarXiv:1210.3149v2 [q-bio.MN] 16 Oct 2014 2 MIC on four synthetic datasets generated by GeneNetWeaver (GNW) [36] following the guidelines of the DREAM Challenge [37], and a transcriptomic dataset on the expression response of human T cells to phorbol 12-myristate 13-acetate (PMA)…”
MethodInfluentialStructure identification for gene regulatory networks via linearization and robust state estimation
Xiong, J., & Zhou, T. (2014) · Automatica · doi:10.1016/j.automatica.2014.08.003 · Cited by 18
“In Prill et al. (2010), it is pointed out that in GRN structure identification, the most well encountered false positive errors are the so called shortcut edge and the so called co-regulated edges.”
BackgroundReaction network realizations of rational biochemical systems and their structural properties
Gábor, A., Hangos, K. M., Banga, J. R., & Szederkényi, G. (2015) · Journal of Mathematical Chemistry · doi:10.1007/s10910-015-0511-9 · Cited by 17
Neural Additive Vector Autoregression Models for Causal Discovery in Time Series
Bussmann, B., Nys, J., & Latré, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88942-5_35 · Cited by 59
DREAMTools: a Python package for scoring collaborative challenges
Cokelaer, T., Bansal, M., Bare, C., Bilal, E., Bot, B. M., Neto, E. C., Eduati, F., de la Fuente, A., Gönen, M., Hill, S. M., et al. (2016) · F1000Research · doi:10.12688/f1000research.7118.2 · Cited by 18
“The task was to identify each of the measured proteins from among the seven molecular species: complex, kinase, phosphorylated complex, phosphorylated protein, protein, phosphatase, and activated phosphatase) 3 .”
InfluentialSelf-citationDREAMTools: a Python package for scoring collaborative challenges
Cokelaer, T., Bansal, M., Bare, C., Bilal, E., Bot, B. M., Neto, E. C., Eduati, F., Gönen, M., Hill, S. M., Hoff, B., et al. (2015) · F1000Research · doi:10.12688/f1000research.7118.1 · Cited by 18
Self-citationFast and accurate inference of gene regulatory networks through robust precision matrix estimation
Passemiers, A., Moreau, Y., & Raimondi, D. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac178 · Cited by 19
GRNbenchmark - a web server for benchmarking directed gene regulatory network inference methods
Seçilmiş, D., Hillerton, T., & Sonnhammer, E. L. L. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac377 · Cited by 17
“We observe that the winner of all methods is Z-score at high noise, while other methods such as least squares and LASSO may outperform it at medium and low noise levels.”
MethodInfluentialStigler, B., & Chamberlin, H. (2012) · BMC Systems Biology · doi:10.1186/1752-0509-6-77 · Cited by 17
“Such approaches have been termed “ pipelines, ” and they exhibit improved performance over individual methods alone [21].”
BackgroundElastic-Net Copula Granger Causality for Inference of Biological Networks
Furqan, M. S., & Siyal, M. Y. (2016) · PLoS ONE · doi:10.1371/journal.pone.0165612 · Cited by 17
wpLogicNet: logic gate and structure inference in gene regulatory networks
Malekpour, S. A., Shahdoust, M., Aghdam, R., & Sadeghi, M. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad072 · Cited by 16
Melas, I. N., Mitsos, A., Messinis, D. E., Weiß, T. S., Sáez-Rodríguez, J., & Alexopoulos, L. G. (2012) · Molecular BioSystems · doi:10.1039/c2mb05482e · Cited by 15
Self-citationLoh, P., Tucker, G., & Berger, B. (2011) · PLoS ONE · doi:10.1371/journal.pone.0029095 · Cited by 19
“It is difficult to objectively measure progress on algorithmic challenges without standard benchmarks; within this context, the Dialogue for Reverse Engineering Assessments and Methods (DREAM) initiative [16] aims to provide a fair comparison of methods and a clear sense of the reliability of the models.”
Li, P., Gong, P., Li, H., Perkins, E. J., Wang, N., & Zhang, C. (2014) · EURASIP Journal on Bioinformatics and Systems Biology · doi:10.1186/s13637-014-0012-3 · Cited by 15
An effective framework for reconstructing gene regulatory networks from genetical genomics data
Flassig, R., Heise, S., Sundmacher, K., & Klamt, S. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts679 · Cited by 18
A Boolean Approach to Linear Prediction for Signaling Network Modeling
Eduati, F., Corradin, A., Camillo, B. D., & TOFFOLO, G. M. (2010) · PLoS ONE · doi:10.1371/journal.pone.0012789 · Cited by 15
Reverse Engineering Validation using a Benchmark Synthetic Gene Circuit in Human Cells
Kang, T., White, J. T., Xie, Z., Benenson, Y., Sontag, E. D., & Bleris, L. G. (2013) · ACS Synthetic Biology · doi:10.1021/sb300093y · Cited by 16
Predicting dynamic signaling network response under unseen perturbations
Zhu, F., & Guan, Y. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu382 · Cited by 14
“As quoted from the DREAM 3 report (Prill et al. 2010), " Overall, a handful of best-performer teams were identified, while a majority of teams made predictions that were equivalent to random " .”
BackgroundKnowledge of the perturbation design is essential for accurate gene regulatory network inference
Seçilmiş, D., Hillerton, T., Tjärnberg, A., Nelander, S., Nordling, T. E. M., & Sonnhammer, E. L. L. (2022) · Scientific Reports · doi:10.1038/s41598-022-19005-x · Cited by 19
Segura-Ortiz, A., García-Nieto, J., Aldana-Montes, J. F., & Navas‐Delgado, I. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106653 · Cited by 17
STEME: A Robust, Accurate Motif Finder for Large Data Sets
Reid, J. E., & Wernisch, L. (2014) · PLoS ONE · doi:10.1371/journal.pone.0090735 · Cited by 16
“Like many other tasks in computational biology such as protein interaction prediction [23] and gene regulatory network inference [24], the lack of a gold standard makes the evaluation of motif finding algorithms difficult at best.”
Setting the Standards for Signal Transduction Research
Sáez-Rodríguez, J., Alexopoulos, L. G., & Stolovitzky, G. A. (2011) · Science Signaling · doi:10.1126/scisignal.2001844 · Cited by 14
Self-citationEstimating Linear and Nonlinear Gene Coexpression Networks by Semiparametric Neighborhood Selection
Kontio, J., Rinta-aho, M., & Sillanpää, M. J. (2020) · Genetics · doi:10.1534/genetics.120.303186 · Cited by 14
A Relative Variation-Based Method to Unraveling Gene Regulatory Networks
Wang, Y., & Zhou, T. (2012) · PLoS ONE · doi:10.1371/journal.pone.0031194 · Cited by 13
“Although these estimates may be crude, they are widely adopted in GRN topology estimation and are capable of leading to a good network estimate [16,18].”
BackgroundMethodResultInfluentialTeamwork: Improved eQTL Mapping Using Combinations of Machine Learning Methods
Ackermann, M., Clément‐Ziza, M., Michaelson, J. J., & Beyer, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0040916 · Cited by 15
“In the dream challenge, the area under the Receiver Operator Characteristic (ROC) curve (AUROC) and the area under the precision-recall curve (AUPR) were used to evaluate the performance of the prediction methods [18,19].”
Method
Metadata
- Authors
- Prill, R. J., Marbach, D., Saez-Rodriguez, J., Sorger, P. K., Alexopoulos, L. G., Xue, X., Clarke, N. D., Altan-Bonnet, G., & Stolovitzky, G.
- Affiliations
- IBM (United States) (US)
- École Polytechnique Fédérale de Lausanne (CH)
- Harvard University (US)
- National Technical University of Athens (GR)
- Genome Institute of Singapore (SG)
- Corresponding author
- Stolovitzky, Gustavo
- ORCID
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2010-02-23
- Volume
- 5
- Issue
- 2
- Pages
- e9202
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC2826397
- OpenAlex
- W2143076232
- Cited by
- 439
- References
- 33
- Citations by year
- 2026: 16 · 2025: 16 · 2024: 15 · 2023: 11 · 2022: 21 · 2021: 18 · 2020: 17 · 2019: 17 · 2018: 28 · 2017: 27 · 2016: 23 · 2015: 50 · 2014: 40 · 2013: 38 · 2012: 47
- FWCI
- 18.23
- Type
- journal article
- Language
- en
- Update notices
- correction (2010-03-09)
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- National University of Singapore (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- DSpace - NTUA (National Technical University of Athens) (submittedVersion, other-oa)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- Funders
- National Science Foundation (GM68762, 0848030)
- National Institutes of Health (3p50gm068762-07s2, NIH P50, GM68762)
- National Cancer Institute (U54CA112967)
- National Institute of General Medical Sciences (P50 GM68762, P50-GM068762)
- National Institute of Allergy and Infectious Diseases (R56AI083408, AI083408, R01AI083408)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Microbial Metabolic Engineering and Bioproduction
- Keywords
- network inference, reverse engineering, gene expression prediction, ensemble forecasting, model validation
- MeSH terms
- Algorithms, Animals, Humans, Models, Biological, Software, Reproducibility of Results, Signal Transduction, Cluster Analysis, Computational Biology, Gene Expression Profiling, Protein Interaction Mapping, Systems Biology, Gene Regulatory Networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.