Towards a Rigorous Assessment of Systems Biology Models: The DREAM3 Challenges
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Systems biology has embraced computational modeling in response to the quantitative nature and increasing scale of contemporary data sets. The onslaught of data is accelerating as molecular profiling technology evolves. The Dialogue for Reverse Engineering Assessments and Methods (DREAM) is a community effort to catalyze discussion about the design, application, and assessment of systems biology models through annual reverse-engineering challenges. METHODOLOGY AND PRINCIPAL FINDINGS: We describe our assessments of the four challenges associated with the third DREAM conference which came to be known as the DREAM3 challenges: signaling cascade identification, signaling response prediction, gene expression prediction, and the DREAM3 in silico network challenge. The challenges, based on anonymized data sets, tested participants in network inference and prediction of measurements. Forty teams submitted 413 predicted networks and measurement test sets. Overall, a handful of best-performer teams were identified, while a majority of teams made predictions that were equivalent to random. Counterintuitively, combining the predictions of multiple teams (including the weaker teams) can in some cases improve predictive power beyond that of any single method. CONCLUSIONS: DREAM provides valuable feedback to practitioners of systems biology modeling. Lessons learned from the predictions of the community provide much-needed context for interpreting claims of efficacy of algorithms described in the scientific literature.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Their results show that, in addition to the obstacles identified by Prill et al. [94] (lack of data and deficiencies in the inference algorithms), we must be also aware of fundamental problems related to the uniqueness and distinguishability of these networks (even for the utopian case of fully observed networks with no noise).”
From Reverse engineering and identification in systems biology: strategies, perspectives and challenges (2014) · cites this paper as [94]
In the citing paper
“The organizers identify two major hurdles to be surmounted: lack of data and deficiencies in the inference algorithms.”
From Reverse engineering and identification in systems biology: strategies, perspectives and challenges (2014) · cites this paper as [94]
In the citing paper
“In other words, reliable network inference remains an unsolved problem.”
From Reverse engineering and identification in systems biology: strategies, perspectives and challenges (2014) · cites this paper as [94]
In the citing paper
“There are several notable examples where collaborations have been very successful, such as SBML [238], BioModels Database [239] or the DREAM challenges [94].”
From Reverse engineering and identification in systems biology: strategies, perspectives and challenges (2014) · cites this paper as [94]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Exploration, normalization, and summaries of high density oligonucleotide array probe level data
Irizarry, R. A. (2003) · Biostatistics · doi:10.1093/biostatistics/4.2.249 · Cited by 10,603 · cited 1×
“Measurements were normalized using the RMA algorithm [18] within the commercial software package, GeneSpring.”
MethodJournal of the Royal Statistical Society (B)
Fildes, R. (1993) · International Journal of Forecasting · doi:10.1016/0169-2070(93)90088-5 · Cited by 3,437
Margolin, A. A., Nemenman, I., Basso, K., Wiggins, C., Stolovitzky, G., Favera, R. D., & Califano, A. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-s1-s7 · Cited by 2,420 · cited 1×
“…be applied off-the-shelf, a strategy for guarding against bogus predictions is to employ a wide variety of algorithms that train on different features of the data (e.g., correlation [28], mutual information [29–32], synergy [33], etc.), then merge the results to gain confidence in the predictions.”
ResultSelf-citationMargolin, A. A., Nemenman, I. M., Basso, K., Wiggins, C. H., Stolovitzky, G. A., Favera, R. D., & Califano, A. (2006) · BMC Bioinformatics · doi:10.7916/d8g15zbp · Cited by 2,246
Self-citationReverse engineering of regulatory networks in human B cells
Basso, K., Margolin, A. A., Stolovitzky, G., Klein, U., Dalla-Favera, R., & Califano, A. (2005) · Nature Genetics · doi:10.1038/ng1532 · Cited by 1,404 · cited 1×
“, correlation [28], mutual information [29–32], synergy [33], etc.”
BackgroundSelf-citationBUTTE, A. J., & KOHANE, I. S. (1999) · Biocomputing 2000 · doi:10.1142/9789814447331_0040 · Cited by 945 · cited 1×
“, correlation [28], mutual information [29–32], synergy [33], etc.”
BackgroundNatarajan, K., Meyer, M. R., Jackson, B. M., Slade, D., Roberts, C., Hinnebusch, A. G., & Marton, M. J. (2001) · Molecular and Cellular Biology · doi:10.1128/mcb.21.13.4347-4368.2001 · Cited by 748 · cited 1×
“Expression levels were assayed separately in the three mutant strains and in the wild type strain at times 0, 10, 20, 30, 45, 60, 90 and 120 minutes following the addition of 3-aminotriazole (3AT) as described [17].”
MethodButte, A. J., Tamayo, P., Slonim, D., Golub, T. R., & Kohane, I. S. (2000) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.220392197 · Cited by 605 · cited 1×
“…be applied off-the-shelf, a strategy for guarding against bogus predictions is to employ a wide variety of algorithms that train on different features of the data (e.g., correlation [28], mutual information [29–32], synergy [33], etc.), then merge the results to gain confidence in the predictions.”
ResultBonneau, R., Reiss, D. J., Shannon, P., Facciotti, M., Hood, L., Baliga, N. S., & Thorsson, V. (2006) · Genome Biology · doi:10.1186/gb-2006-7-5-r36 · Cited by 510
Research works printed but not linked: 1
- [15]Alexopoulos L Saez-Rodriguez J Cosgrove B Lauffenburger D PK S 2009 Networks reconstructed from cell response data reveal profound dif ferences in signaling by tolllike receptors and nf-?b in normal and transformed human hepatocytes. Under review
Cited by
Inferring Regulatory Networks from Expression Data Using Tree-Based Methods
Huynh-Thu, V. A., Irrthum, A., Wehenkel, L., & Geurts, P. (2010) · PLoS ONE · doi:10.1371/journal.pone.0012776 · Cited by 2,131
“DREAM4 Datasets The DREAM (for ‘‘Dialogue for Reverse Engineering Assessments and Methods’’) initiative organizes an annual reverse engineering competition called the DREAM challenge [34–37].”
BackgroundWisdom of crowds for robust gene network inference
The DREAM5 Consortium, Marbach, D., Costello, J. C., Küffner, R., Vega, N. M., Prill, R. J., Camacho, D. M., Allison, K. R., Kellis, M., Collins, J. J., et al. (2012) · Nature Methods · doi:10.1038/nmeth.2016 · Cited by 1,667
Self-citationA Validated Regulatory Network for Th17 Cell Specification
Ciofani, M., Madar, A., Galan, C., Sellars, M., Mace, K., Pauli, F., Agarwal, A., Huang, W., Parkurst, C. N., Muratet, M., et al. (2012) · Cell · doi:10.1016/j.cell.2012.09.016 · Cited by 1,160
Critical assessment of automated flow cytometry data analysis techniques
The FlowCAP Consortium, Aghaeepour, N., The DREAM Consortium, Finak, G., Hoos, H., Mosmann, T. R., Brinkman, R., Gottardo, R., & Scheuermann, R. H. (2013) · Nature Methods · doi:10.1038/nmeth.2365 · Cited by 598
GeneNetWeaver: in silico benchmark generation and performance profiling of network inference methods
Schaffter, T., Marbach, D., & Floreano, D. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr373 · Cited by 535
“…generated using random graphs capture (Van den Bulcke et al. , 2006) and do generally not display important properties such as modularity (Ravasz et al. , 2002) or occurrences of network motifs, which are statistically overrepresented regulatory theory, which may be difficult to justify…”
Self-citationComparison of co-expression measures: mutual information, correlation, and model based indices
Song, L., Langfelder, P., & Horvath, S. (2012) · BMC Bioinformatics · doi:10.1186/1471-2105-13-328 · Cited by 460
“For example, the DREAM (Dialogue for Reverse Engineering Assessments and Methods) project has repeatedly tackled this problem [41-43].”
BackgroundEvaluation of methods for modeling transcription factor sequence specificity
DREAM5 Consortium, Weirauch, M. T., Cote, A., Norel, R., Annala, M., Zhao, Y., Riley, T. R., Saez-Rodriguez, J., Cokelaer, T., Vedenko, A., et al. (2013) · Nature Biotechnology · doi:10.1038/nbt.2486 · Cited by 381
“The DREAM5 TF challenge 21-23 formed the original basis for the analyses presented here.”
BackgroundSelf-citationTank, A., Covert, I., Foti, N., Shojaie, A., & Fox, E. B. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3065601 · Cited by 282
“Second, we compare our cMLP and cLSTM models with existing Granger causality approaches [33], [34] on the difficult DREAM3 gene regulatory network recovery benchmark datasets [35] and find that our methods outperform a wide set of competitors across all five datasets.”
MethodCkmeans.1d.dp: Optimal k-means Clustering in One Dimension by Dynamic Programming
Wang, H., & Song, M. (2011) · The R Journal · doi:10.32614/rj-2011-015 · Cited by 374
“We applied our Ckmeans.1d.dp to quan-tize biological data in the DREAM Challenges (Prill et al., 2010) for qualitative dynamic modeling using generalized logical networks (Song et al., 2009).”
MethodComprehensive benchmarking and ensemble approaches for metagenomic classifiers
McIntyre, A. B. R., Ounit, R., Afshinnekoo, E., Prill, R. J., Hénaff, E., Alexander, N., Minot, S. S., Danko, D., Foox, J., Ahsanuddin, S., et al. (2017) · Genome Biology · doi:10.1186/s13059-017-1299-7 · Cited by 342
“Variations on this method of combining multiple ranked lists into a single list have been shown to effectively mitigate the uncertainty about which tool(s) are the most accurate on a particular dataset [70, 71] and for complex samples [72].”
MethodSelf-citationA community computational challenge to predict the activity of pairs of compounds
Community, N.-D., Bansal, M., Yang, J., Karan, C., Menden, M. P., Costello, J. C., Tang, H., Xiao, G., Li, Y., Allen, J., et al. (2014) · Nature Biotechnology · doi:10.1038/nbt.3052 · Cited by 321
“( b ) Average and standard error over the 1,000 splits shown in a of the PC-index as computed in set S3 of individual teams (blue) and aggregates of the top-performing teams (red).”
MethodSelf-citationSTATISTICAL METHODS FOR THE ANALYSIS OF HIGH-THROUGHPUT METABOLOMICS DATA
Bartel, J., Krumsiek, J., & Theis, F. J. (2013) · Computational and Structural Biotechnology Journal · doi:10.5936/csbj.201301009 · Cited by 297
Greenfield, A., Madar, A., Ostrer, H., & Bonneau, R. (2010) · PLoS ONE · doi:10.1371/journal.pone.0013397 · Cited by 255
“The topology predictions were evaluated by area under the precision recall curve (AUPR) [5].”
MethodHauser, A., & Bühlmann, P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1104.2808 · Cited by 226
“3 DREAM4 Challenge We also measured the performance of GIES on synthetic gene expression data sets from the DREAM4 in silico challenge (Marbach et al., 2010; Prill et al., 2010).”
MethoddynGENIE3: dynamical GENIE3 for the inference of gene networks from time series expression data
Huynh-Thu, V. A., & Geurts, P. (2018) · Scientific Reports · doi:10.1038/s41598-018-21715-0 · Cited by 245
“To evaluate such a ranking independently of the choice of a specific threshold, we use the precision-recall (PR) curve and the area under this curve (AUPR), as suggested by the DREAM consortium 7,23–25 .”
MethodInferring causal molecular networks: empirical assessment through a community-based effort
The HPN-DREAM Consortium, Hill, S. M., Heiser, L. M., Cokelaer, T., Unger, M., Nesser, N. K., Carlin, D. E., Zhang, Y., Sokolov, A., Paull, E. O., et al. (2016) · Nature Methods · doi:10.1038/nmeth.3773 · Cited by 231
Self-citationReverse engineering and identification in systems biology: strategies, perspectives and challenges
Villaverde, A. F., & Banga, J. R. (2014) · Journal of The Royal Society Interface · doi:10.1098/rsif.2013.0505 · Cited by 229
“There are several notable examples where collaborations have been very successful, such as SBML [238], BioModels Database [239] or the DREAM challenges [94].”
BackgroundComputational methods for Gene Regulatory Networks reconstruction and analysis: A review
Delgado, F. M., & Gómez-Vela, F. (2019) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2018.10.006 · Cited by 206
“[127], an evaluation of GRN inference methods performance is carried out highlighting best strategies.”
MethodDirect measurement of DNA affinity landscapes on a high-throughput sequencing instrument
Nutiu, R., Friedman, R. C., Luo, S., Khrebtukova, I., Silva, D., Li, R., Zhang, L., Schroth, G. P., & Burge, C. B. (2011) · Nature Biotechnology · doi:10.1038/nbt.1882 · Cited by 194
Supervised, semi-supervised and unsupervised inference of gene regulatory networks
Maetschke, S. R., Madhamshettiwar, P. B., Davis, M. J., & Ragan, M. A. (2014) · Briefings in Bioinformatics · doi:10.1093/bib/bbt034 · Cited by 167
“[7], ‘The vast majority of the teams’ predictions were statistically equivalent to random guesses’.”
BackgroundMethodInfluentialAn Automated Model Test System for Systematic Development and Improvement of Gene Expression Models
Reis, A. C., & Salis, H. M. (2020) · ACS Synthetic Biology · doi:10.1021/acssynbio.0c00394 · Cited by 166
Crowdsourcing biomedical research: leveraging communities as innovation engines
Saez-Rodriguez, J., Costello, J. C., Friend, S. H., Kellen, M. R., Mangravite, L., Meyer, P., Norman, T., & Stolovitzky, G. (2016) · Nature Reviews Genetics · doi:10.1038/nrg.2016.69 · Cited by 165
“Different strategies were used to circumvent this problem: simulated expression data, which enabled systematic evaluation based on the underlying in silico gene networks 40–42,45–”
MethodSelf-citationHow to infer gene networks from expression profiles, revisited
Penfold, C. A., & Wild, D. L. (2011) · Interface Focus · doi:10.1098/rsfs.2011.0053 · Cited by 162
Madhamshettiwar, P. B., Maetschke, S. R., Davis, M. J., Reverter, A., & Ragan, M. A. (2012) · Genome Medicine · doi:10.1186/gm340 · Cited by 158
“Finally, we compare the best-performing unsupervised methods with the supervised method using simulated datasets obtained from the DREAM3 and DREAM4 competitions [15,29] and datasets generated using the SynTReN software [30].”
BackgroundMethodInfluentialModeling Influenza Virus Infection: A Roadmap for Influenza Research
Boianelli, A., Nguyễn, V. K., Ebensen, T., Schulze, K., Wilk, E., Sharma, N., Stegemann‐Koniszewski, S., Bruder, D., Toapanta, F. R., Guzmán, C. A., et al. (2015) · Viruses · doi:10.3390/v7102875 · Cited by 172
“The structural and practical identifiability are necessary in mathematical models to reach significant predictions [66–71].”
BackgroundSaez-Rodriguez, J., Alexopoulos, L. G., Zhang, M., Morris, M. K., Lauffenburger, D. A., & Sorger, P. K. (2011) · Cancer Research · doi:10.1158/0008-5472.can-10-4453 · Cited by 147
“Continued development of CellNOpt on our part, as well as the creation of alternative modeling approaches fostered by efforts such as DREAM (41), is likely to improve the efficiency and accuracy of network inference from biochemical data.”
Self-citationGreenfield, A., Hafemeister, C., & Bonneau, R. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt099 · Cited by 141
Navigating cancer network attractors for tumor-specific therapy
Creixell, P., Schoof, E. M., Erler, J. T., & Linding, R. (2012) · Nature Biotechnology · doi:10.1038/nbt.2345 · Cited by 153
Combining tree-based and dynamical systems for the inference of gene regulatory networks
Huynh-Thu, V. A., & Sanguinetti, G. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btu863 · Cited by 126
“To evaluate such a ranking independently of the choice of a specific threshold, we use the precision-recall (PR) curve and the area under this curve (AUPR).”
MethodSpearheading future omics analyses using dyngen, a multi-modal simulator of single cells
Cannoodt, R., Saelens, W., Deconinck, L., & Saeys, Y. (2021) · Nature Communications · doi:10.1038/s41467-021-24152-2 · Cited by 124
Margolin, A. A., Bilal, E., Huang, E., Norman, T. C., Ottestad, L., Mecham, B. H., Sauerwine, B., Kellen, M. R., Mangravite, L. M., Furia, M. D., et al. (2013) · Science Translational Medicine · doi:10.1126/scitranslmed.3006112 · Cited by 121
Self-citationTarca, A. L., Lauria, M., Unger, M., Bilal, E., Boué, S., Dey, K. K., Hoeng, J., Koeppl, H., Martin, F. P., Meyer, P., et al. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt492 · Cited by 126
Self-citationSystems-based proteomics to resolve the biology of Alzheimer’s disease beyond amyloid and tau
Rayaprolu, S., Higginbotham, L., Bagchi, P., Watson, C. M., Zhang, T., Levey, A. I., Rangaraju, S., & Seyfried, N. T. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-020-00840-3 · Cited by 118
Inferring gene regulatory networks by ANOVA
Küffner, R., Petri, T., Tavakkolkhah, P., Windhager, L., & Zimmer, R. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts143 · Cited by 135
“In addition, our approach was evaluated as the best performer for the inference of real datasets in the recent DREAM5 blind assessment [http://wiki.c2b2.columbia.edu/ dream/index.php; Marbach et al. (2010); Prill et al. (2010)].”
MethodPartial cross mapping eliminates indirect causal influences
Leng, S., Ma, H., Kurths, J., Lai, Y., Lin, W. J., Aihara, K., & Chen, L. (2020) · Nature Communications · doi:10.1038/s41467-020-16238-0 · Cited by 136
Improving Breast Cancer Survival Analysis through Competition-Based Multidimensional Modeling
Bilal, E., Dutkowski, J., Guinney, J., Jang, I. S., Logsdon, B. A., Pandey, G. S., Sauerwine, B., Shimoni, Y., Vollan, H. K. M., Mecham, B. H., et al. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003047 · Cited by 124
“As previously suggested in the context of the DREAM project [38–41], our finding further reinforces the notion that crowd-sourced collaborative competitions are a powerful framework for developing robust predictive models by training an ensemble model aggregated across diverse strategies employed by participants.”
ResultSelf-citationAlexopoulos, L. G., Saez-Rodriguez, J., Cosgrove, B. D., Lauffenburger, D. A., & Sorger, P. K. (2010) · Molecular & Cellular Proteomics · doi:10.1074/mcp.m110.000406 · Cited by 108
“Subsets of these data have previously been made available as part of the network inference challenges for the Dialogue on Reverse Engineering Assessments and Methods (DREAM), and these data sets are also available upon request.”
BackgroundMethodInfluentialSelf-citationBalaguer, M. A. D. L., Fisher, A., Clark, N. M., Fernández-Espinosa, M. G., Möller, B. K., Weijers, D., Lohmann, J. U., Williams, C., Lorenzo, Ó., & Sozzani, R. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1707566114 · Cited by 119
Vignes, M., Vandel, J., Allouche, D., Ramadan-Alban, N., Cierco-Ayrolles, C., Schiex, T., Mangin, B., & de Givry, S. (2011) · PLoS ONE · doi:10.1371/journal.pone.0029165 · Cited by 116
“The idea of combining results from different methods has already been tested by the DREAM organisers themselves in a previous different DREAM challenge, in what they called ‘‘the community intelligence’’ [36].”
MethodNetwork‐based approaches for understanding gene regulation and function in plants
Ko, D. K., & Brandizzí, F. (2020) · The Plant Journal · doi:10.1111/tpj.14940 · Cited by 112
Dondelinger, F., Lèbre, S., & Husmeier, D. (2012) · Machine Learning · doi:10.1007/s10994-012-5311-x · Cited by 119
“These two measures are widely used in the systems biology literature to quantify the overall network reconstruction accuracy (Prill et al. 2010), with larger values indicating a better prediction performance overall.”
BackgroundA review on computational systems biology of pathogen–host interactions
DurmuÅŸ, S., Çakır, T., Özgür, A., & Guthke, R. (2015) · Frontiers in Microbiology · doi:10.3389/fmicb.2015.00235 · Cited by 117
Verification of systems biology research in the age of collaborative competition
Meyer, P., Alexopoulos, L. G., Bonk, T., Califano, A., Cho, C. R., de la Fuente, A., de Graaf, D., Hartemink, A. J., Hoeng, J., Ivanov, N. V., et al. (2011) · Nature Biotechnology · doi:10.1038/nbt.1968 · Cited by 94
Self-citationNetwork inference in systems biology: recent developments, challenges, and applications
Saint-Antoine, M., & Singh, A. (2020) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2019.12.002 · Cited by 110
“Once each competition is over, the datasets and true network structures are released to the public, and are commonly used for algorithm evaluation [9,10,11].”
MethodThe self‐assessment trap: can we all be better than average?
Norel, R., Rice, J. J., & Stolovitzky, G. (2011) · Molecular Systems Biology · doi:10.1038/msb.2011.70 · Cited by 96
“Furthermore, the effectiveness of a top-performing algorithm can be boosted when its results are aggregated with second and third best performers (Figure 1, and Supplementary Figures S1 and S2; Marbach et al , 2010; Prill et al , 2010).”
Self-citationFrom Knockouts to Networks: Establishing Direct Cause-Effect Relationships through Graph Analysis
Pinna, A., Soranzo, N., & de la Fuente, A. (2010) · PLoS ONE · doi:10.1371/journal.pone.0012912 · Cited by 103
“Method effectiveness was evaluated through the calculations of the Area Under the Receiver Operating Characteristic Curve (AUC(ROC)) and the Area Under the Precision versus Recall Curve (AUC(PvsR)) [20,24], in the same way as was done by the DREAM organizers to evaluate the submitted networks.”
BackgroundMethodGene regulatory networks in plants: learning causality from time and perturbation
Krouk, G., Lingeman, J. M., Marshall‐Colón, A., Coruzzi, G. M., & Shasha, D. E. (2013) · Genome biology · doi:10.1186/gb-2013-14-6-123 · Cited by 101
Barman, S., & Kwon, Y. (2017) · PLoS ONE · doi:10.1371/journal.pone.0171097 · Cited by 91
Network inference and biological dynamics
Oates, C. J., & Mukherjee, S. (2012) · The Annals of Applied Statistics · doi:10.1214/11-aoas532 · Cited by 88
“In particular, the Dialogue for Reverse Engineering Assessments and Methods (DREAM) challenges [Prill et al. (2010)] have provided an opportunity for objective empirical assessment of competing approaches.”
BackgroundInference of complex biological networks: distinguishability issues and optimization-based solutions
Szederkényi, G., Banga, J. R., & Alonso, A. A. (2011) · BMC Systems Biology · doi:10.1186/1752-0509-5-177 · Cited by 78
“Prill et al [27] further explored the issue of intrinsic impediments to network inference, designating identifiability of certain network edges and systematic false positives as the main barriers.”
BackgroundFast Bayesian inference for gene regulatory networks using ScanBMA
Young, W. C., Raftery, A. E., & Yeung, K. Y. (2014) · BMC Systems Biology · doi:10.1186/1752-0509-8-47 · Cited by 94
“DREAM4 In Silico Network Challenge.”
MethodFundamental limitations of network reconstruction from temporal data
Angulo, M. T., Moreno, J. A., Lippner, G., Barabási, A.-Ĺ., & Liu, Y. (2017) · Journal of The Royal Society Interface · doi:10.1098/rsif.2016.0966 · Cited by 93
Karr, J. R., Williams, A. H., Zucker, J., Raue, A., Steiert, B., Timmer, J., Kreutz, C., Wilkinson, S., Allgood, B., Bot, B. M., et al. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004096 · Cited by 79
“DREAM challenges have repeatedly demonstrated the “wisdom of crowds” to produce high-quality methods [32–34].”
BackgroundSelf-citationThomas, H. R., den Broeck, L. V., Spurney, R. J., Sozzani, R., & Frank, M. H. (2021) · The Plant Cell · doi:10.1093/plcell/koab246 · Cited by 85
“Next, we utilized Bayesian inference and regression analyses to expand our understanding of speciesspecific genetic responses, which regulate the conserved process of junction formation (Prill et al., 2010; de Luis Balaguer and Sozzani, 2017; de Luis Balaguer et al., 2017; Clark et al., 2019; Smet et al., 2019).”
BackgroundMethodRitchie, S. C., Watts, S. C., Fearnley, L. G., Holt, K. E., Abraham, G., & Inouye, M. (2016) · Cell Systems · doi:10.1016/j.cels.2016.06.012 · Cited by 90
Meyer, P., Cokelaer, T., Chandran, D., Kim, K., Loh, P.-R., Tucker, G., Lipson, M., Berger, B., Kreutz, C., Raue, A., et al. (2014) · BMC Systems Biology · doi:10.1186/1752-0509-8-13 · Cited by 73
“DREAM results for a diverse set of challenges have recurrently demonstrated the “wisdom of crowds” phenomenon, where aggregation of participants’ results has proven to give robust and top performing results [3,25,26].”
BackgroundResultSelf-citationBhuva, D. D., Cursons, J., Smyth, G. K., & Davis, M. J. (2019) · Genome biology · doi:10.1186/s13059-019-1851-8 · Cited by 76
Causal Discovery from Temporal Data: An Overview and New Perspectives
Gong, C., Zhang, C., Yao, D., Bi, J., Li, W., & Xu, Y. (2024) · ACM Computing Surveys · doi:10.1145/3705297 · Cited by 156
“Specifically, the datasets employed for MTS-CD encompass (i) Lorenz-96 [121], (ii) CMU-MoCap [180], (iii) DREAM-3 [148], (iv) BOLD [170], (v) Financial [104], (vi) MIMIC-III [95], (vii) CIPCaD-Bench [128], (viii) TECDI [110], and (ix) CausalTime [29].”
Inference of gene regulatory networks based on nonlinear ordinary differential equations
Ma, B., Fang, M., & Jiao, X. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa032 · Cited by 80
Network-based approaches for modeling disease regulation and progression
Galindez, G., Sadegh, S., Baumbach, J., Kacprowski, T., & List, M. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2022.12.022 · Cited by 79
Gene regulatory network inference from sparsely sampled noisy data
Aalto, A., Viitasaari, L., Ilmonen, P., Mombaerts, L., & Gonçalves, J. M. (2020) · Nature Communications · doi:10.1038/s41467-020-17217-1 · Cited by 93
A Boolean network inference from time-series gene expression data using a genetic algorithm
Barman, S., & Kwon, Y. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty584 · Cited by 76
NIMEFI: Gene Regulatory Network Inference using Multiple Ensemble Feature Importance Algorithms
Ruyssinck, J., Huynh‐Thu, V. A., Geurts, P., Dhaene, T., Demeester, P. M. A., & Saeys, Y. (2014) · PLoS ONE · doi:10.1371/journal.pone.0092709 · Cited by 74
“In both the DREAM4 in silico 100 multifactorial challenge and the latest DREAM5 network inference challenge [18–20] the overall top performer was the GENIE3 algorithm [21].”
MethodSalleh, F. H. M., Arif, S. M., Zainudin, S., & Firdaus‐Raih, M. (2015) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2015.04.012 · Cited by 67
Chen, Y., Mazlack, L. J., Minai, A. A., & Lu, L. J. (2015) · Applied Soft Computing · doi:10.1016/j.asoc.2015.08.039 · Cited by 65
“[74] R.”
MethodGenomics and data science: an application within an umbrella
Navarro, F. C. P., Mohsen, H., Yan, C., Li, S., Gu, M., Meyerson, W. U., & Gerstein, M. (2019) · Genome biology · doi:10.1186/s13059-019-1724-1 · Cited by 69
“Challenge themes have included “ Genome-Scale Network Inference ” , “ Gene Expression Prediction ” , “ Alternative Splicing ” , and “ in vivo Transcription Factor Binding Site Prediction ” [69].”
BackgroundWhen integration fails: Prokaryote phylogeny and the tree of life
O’Malley, M. A. (2012) · Studies in History and Philosophy of Science Part C Studies in History and Philosophy of Biological and Biomedical Sciences · doi:10.1016/j.shpsc.2012.10.003 · Cited by 68
“sources from one domain to another can produce misleading and even artefactual models (Prill et al., 2010).”
Modelling transcriptional networks in leaf senescence
Penfold, C. A., & Buchanan‐Wollaston, V. (2014) · Journal of Experimental Botany · doi:10.1093/jxb/eru054 · Cited by 68
Analysis of Flow Cytometry Data by Matrix Relevance Learning Vector Quantization
Biehl, M. L., Bunte, K., & Schneider, P. (2013) · PLoS ONE · doi:10.1371/journal.pone.0059401 · Cited by 57
“This challenge was organized in a joint effort by the Dialogue for Reverse Engineering Assessments and Methods (DREAM) project [3–6] and the Flow Cytometry: Critical Assessment of Population Identification Methods (Flow-CAP) initiative [2].”
MethodData- and knowledge-based modeling of gene regulatory networks: an update
Linde, J., Schulze, S., Henkel, S. G., & Guthke, R. (2015) · EXCLI journal · doi:10.17179/excli2015-168 · Cited by 73
“In 2006, Stolovitzky, Monroe and Califano initiated the so-called ‘Dialogue for Reverse Engineering Assessment and Methods’ (www.dreamchallenges.org) (Stolovitzky et al., 2007; Prill et al., 2010; Marbach et al., 2012; Bansal et al., 2014).”
BackgroundLopes, F. M., Martins, D., Barrera, J., & Junior, R. M. C. (2014) · Information Sciences · doi:10.1016/j.ins.2014.02.096 · Cited by 65
“Moreover, it also adopted the area under the precision–recall (AUPR) [51] for this experiment, and provided better results.”
MethodNetwork-based pathway enrichment analysis with incomplete network information
Ma, J., Shojaie, A., & Michailidis, G. C. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw410 · Cited by 62
“The network structure is extracted from the DREAM3 challenge (Prill et al., 2010) corresponding to the Ecoli network (labeled Ecoli 1).”
MethodModeling Gene Regulation Networks Using Ordinary Differential Equations
Cao, J., Qi, X., & Zhao, H. (2011) · Methods in molecular biology · doi:10.1007/978-1-61779-400-1_12 · Cited by 62
Meyer, P., Hoeng, J., Rice, J. J., Norel, R., Sprengel, J., Stolle, K., Bonk, T., Corthesy, S., Royyuru, A. K., Peitsch, M. C., et al. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts116 · Cited by 52
Self-citationDemkow, U. A., & Wolańczyk, T. (2017) · Translational Psychiatry · doi:10.1038/tp.2017.106 · Cited by 59
Fu, Y., Jarboe, L. R., & Dickerson, J. A. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-233 · Cited by 58
“In the field well known conference on Dialogue for Reverse Engineering Assessments and Methods (DREAM) [19], many winning algorithms are based on CLR.”
MethodNonparametric Bayesian inference for perturbed and orthologous gene regulatory networks
Penfold, C. A., Buchanan‐Wollaston, V., Denby, K., & Wild, D. L. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts222 · Cited by 52
Rapid innovation in ChIP-seq peak-calling algorithms is outdistancing benchmarking efforts
Szalkowski, A. M., & Schmid, C. D. (2010) · Briefings in Bioinformatics · doi:10.1093/bib/bbq068 · Cited by 59
“Given the magnitude of such a benchmarking study, a community effort possibly in the context of the popular DREAM challenges [34] appear to represent an optimal framework.”
BackgroundBellot, P., Olsen, C., Salembier, P., Oliveras, A., & Meyer, P. E. (2015) · BMC Bioinformatics · doi:10.1186/s12859-015-0728-4 · Cited by 59
“…the final confidence score assigned between genes i and j is computed as the geometric mean of the scores obtained between gene i and j and vice versa: Zscore Zscore [19] is a method that assumes interventional data, more concretely knockout experiments that lead to a change in other genes.”
MethodNoh, H., Shoemaker, J. E., & Gunawan, R. (2018) · Nucleic Acids Research · doi:10.1093/nar/gkx1314 · Cited by 58
“9 positive rate, following the procedure adopted in DREAM challenges (40-41).”
Method2-hydr_Ensemble: Lysine 2-hydroxyisobutyrylation identification with ensemble method
Bao, W., Yang, B., & Chen, B. (2021) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2021.104351 · Cited by 67
Inferring Nonlinear Gene Regulatory Networks from Gene Expression Data Based on Distance Correlation
Guo, X., Zhang, Y., Hu, W., Tan, H., & Wang, X. (2014) · PLoS ONE · doi:10.1371/journal.pone.0087446 · Cited by 59
“To this end, the DREAM (Dialogue for Reverse Engineering Assessments and Methods) program and its conference series are devoted to encourage researchers to investigate novel powerful methods [4-6].”
BackgroundCUTS+: High-Dimensional Causal Discovery from Irregular Time-Series
Cheng, Y., Li, L., Xiao, T., Li, Z., Suo, J., He, K., & Dai, Q. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.29034 · Cited by 116
Information-Theoretic Inference of Gene Networks Using Backward Elimination
Meyer, P. E., Marbach, D., Roy, S., & Kellis, M. (2010) · ORBi (University of Liège) · Cited by 58
Self-citationPSI: A Comprehensive and Integrative Approach for Accurate Plant Subcellular Localization Prediction
Liu, L., Zhang, Z., Mei, Q., & Chen, M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0075826 · Cited by 57
“According to the cross validation with area under the ROC curve (AUROC) [32], the area under the precision vs.”
InfluentialAghdam, R., Ganjali, M., Zhang, X., & Eslahchi, C. (2014) · Molecular BioSystems · doi:10.1039/c4mb00413b · Cited by 52
Operating principles of tristable circuits regulating cellular differentiation
Jia, D., Jolly, M. K., Harrison, W., Boareto, M., Ben‐Jacob, E., & Levine, H. (2017) · Physical Biology · doi:10.1088/1478-3975/aa6f90 · Cited by 54
Mall, R., Cerulo, L., Garofano, L., Frattini, V., Kunji, K. B., Bensmail, H., Sabedot, T. S., Noushmehr, H., Lasorella, A., Iavarone, A., et al. (2018) · Nucleic Acids Research · doi:10.1093/nar/gky015 · Cited by 49
“…of the proposed RGBM models using LS-Boost and LAD-Boost as core models and RGENIE using RF as core model on universally accepted benchmark networks of 100 or more genes from the DREAM3, DREAM4 and DREAM5 challenges (55–57) and compared them with several state-of-the-art GRN inference methods.”
MethodReview and assessment of Boolean approaches for inference of gene regulatory networks
Pušnik, Ž., Mraz, M., Zimic, N., & Moškon, M. (2022) · Heliyon · doi:10.1016/j.heliyon.2022.e10222 · Cited by 51
Narendra, V. K., Lytkin, N. I., Aliferis, C., & Statnikov, A. (2010) · Genomics · doi:10.1016/j.ygeno.2010.10.003 · Cited by 57
“This led to the formation of the DREAM project—a series of four competitions designed to assess the accuracy of network reverse-engineering [40,45,46].”
MethodAderhold, A., Husmeier, D., Lennon, J. J., Beale, C. M., & Smith, V. A. (2012) · Ecological Informatics · doi:10.1016/j.ecoinf.2012.05.002 · Cited by 58
“…the ROC (AUROC) and precision-recall (AUPRC) curves (Davis and Goadrich, 2006), which are two measures widely used in the systems biology literature to quantify the overall network reconstruction accuracy (Prill et al., 2010), with larger values indicating a better prediction performance overall.”
BackgroundMethodInfluentialNetwork Reconstruction Using Nonparametric Additive ODE Models
Henderson, J., & Michailidis, G. C. (2014) · PLoS ONE · doi:10.1371/journal.pone.0094003 · Cited by 64
“In addition to the computational models described above, we also evaluated the NeRDS methodology on the 10-and 100-node networks from DREAM 3, challenge 4 [8,42,43].”
Mapping functional transcription factor networks from gene expression data
Haynes, B. C., Maier, E. J., Kramer, M. H., Wang, P. I., Brown, H., & Brent, M. R. (2013) · Genome Research · doi:10.1101/gr.150904.112 · Cited by 54
“An annual community effort to evaluate such algorithms, known as DREAM (Dialogue for Reverse Engineering Assessments and Methods), has been held six times (Stolovitzky et al. 2009; Prill et al. 2010).”
MethodBoutros, P. C., Ewing, A. D., Ellrott, K., Norman, T., Dang, K. K., Hu, Y., Kellen, M., Suver, C., Bare, J. C., Stein, L. D., et al. (2014) · Nature Genetics · doi:10.1038/ng.2932 · Cited by 44
Self-citationAssessment of Network Inference Methods: How to Cope with an Underdetermined Problem
Siegenthaler, C., & Gunawan, R. (2014) · PLoS ONE · doi:10.1371/journal.pone.0090481 · Cited by 47
“(ZIP) Figure S1 Flowchart for calculation of the AUROC and AUPR in the new assessment.”
BackgroundInfluentialOperator-valued kernel-based vector autoregressive models for network inference
Lim, N., d’Alché–Buc, F., Auliac, C., & Michailidis, G. C. (2014) · Machine Learning · doi:10.1007/s10994-014-5479-3 · Cited by 59
Functional association networks as priors for gene regulatory network inference
Studham, M. E., Tjärnberg, A., Nordling, T. E. M., Nelander, S., & Sonnhammer, E. L. L. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu285 · Cited by 43
“Contact: matthew.studham@scilifelab.se Supplementary information: Supplementary data are available at Bioinformatics online.”
BackgroundENNET: inferring large gene regulatory networks from expression data using gradient boosting
Sławek, J., & Arodź, T. (2013) · BMC Systems Biology · doi:10.1186/1752-0509-7-106 · Cited by 49
“The results of all the competing meth-ods except those that are aimed at multifactorial problems are summarized in Table 2 DREAM4 challenge [15,53,54] was posted one year after DREAM3 challenge.”
BackgroundMethodInfluentialModeling Signaling Networks Using High-throughput Phospho-proteomics
Terfve, C., & Sáez-Rodríguez, J. (2011) · Advances in experimental medicine and biology · doi:10.1007/978-1-4419-7210-1_2 · Cited by 52
“…combinations of stimuli and inhibitors of a signaling pathway (based on measurements of the same players under different combinations of the same stimuli/inhibitors), methods that performed the best used a statistical approach that did not rely explicitly on an underlying signaling network [57].”
MethodToward better benchmarking: challenge-based methods assessment in cancer genomics
Boutros, P. C., Margolin, A. A., Stuart, J. M., Califano, A., & Stolovitzky, G. (2014) · Genome Biology · doi:10.1186/s13059-014-0462-7 · Cited by 43
“Challenge-based benchmarking efforts, such as CASP [8-10], CAFA [11] and DREAM [12,13], among others (Table 1), offer a robust framework for algorithm evaluation.”
MethodSelf-citation
Metadata
- Authors
- Prill, R. J., Marbach, D., Saez-Rodriguez, J., Sorger, P. K., Alexopoulos, L. G., Xue, X., Clarke, N. D., Altan-Bonnet, G., & Stolovitzky, G.
- Affiliations
- IBM (United States) (US)
- École Polytechnique Fédérale de Lausanne (CH)
- Harvard University (US)
- National Technical University of Athens (GR)
- Genome Institute of Singapore (SG)
- Corresponding author
- Stolovitzky, Gustavo
- ORCID
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2010-02-23
- Volume
- 5
- Issue
- 2
- Pages
- e9202
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC2826397
- OpenAlex
- W2143076232
- Cited by
- 439
- References
- 33
- Citations by year
- 2026: 16 · 2025: 16 · 2024: 15 · 2023: 11 · 2022: 21 · 2021: 18 · 2020: 17 · 2019: 17 · 2018: 28 · 2017: 27 · 2016: 23 · 2015: 50 · 2014: 40 · 2013: 38 · 2012: 47
- FWCI
- 18.23
- Type
- journal article
- Language
- en
- Update notices
- correction (2010-03-09)
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- National University of Singapore (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- DSpace - NTUA (National Technical University of Athens) (submittedVersion, other-oa)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- Funders
- National Science Foundation (GM68762, 0848030)
- National Institutes of Health (3p50gm068762-07s2, NIH P50, GM68762)
- National Cancer Institute (U54CA112967)
- National Institute of General Medical Sciences (P50 GM68762, P50-GM068762)
- National Institute of Allergy and Infectious Diseases (R56AI083408, AI083408, R01AI083408)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Microbial Metabolic Engineering and Bioproduction
- Keywords
- network inference, reverse engineering, gene expression prediction, ensemble forecasting, model validation
- MeSH terms
- Algorithms, Animals, Humans, Models, Biological, Software, Reproducibility of Results, Signal Transduction, Cluster Analysis, Computational Biology, Gene Expression Profiling, Protein Interaction Mapping, Systems Biology, Gene Regulatory Networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.