A benchmark for methods in reverse engineering and model discrimination: problem formulation and solutions
Open accessAbstract
A benchmark problem is described for the reconstruction and analysis of biochemical networks given sampled experimental data. The growth of the organisms is described in a bioreactor in which one substrate is fed into the reactor with a given feed rate and feed concentration. Measurements for some intracellular components are provided representing a small biochemical network. Problems of reverse engineering, parameter estimation, and identifiability are addressed. The contribution mainly focuses on the problem of model discrimination. If two or more model variants describe the available experimental data, a new experiment must be designed to discriminate between the hypothetical models. For the problem presented, the feed rate and feed concentration of a bioreactor system are available as control inputs. To verify calculated input profiles an interactive Web site (http://www.sysbio.de/projects/benchmark/) is provided. Several solutions based on linear and nonlinear models are discussed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent literature is correcting the overlooking of identifiability in systems biology modelling studies.
“Fortunately, recent literature is correcting this (e.g. [129–140]).”
From Reverse engineering and identification in systems biology: strategies, perspectives and challenges (2014) · cites this paper as [129]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Numerical recipes in C: the art of scientific computing
(1993) · Choice Reviews Online · doi:10.5860/choice.30-5638 · Cited by 17,933 · see citing works
System Identification: Theory for the User
Lennart Ljung, A. (1987) · Cited by 17,690 · see citing works
Numerical Recipes in C: The Art of Scientific Computing.
Birchenhall, C. R., Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · The Economic Journal · doi:10.2307/2234664 · Cited by 4,355 · see citing works
Integrated Genomic and Proteomic Analyses of a Systematically Perturbed Metabolic Network
Ideker, T., Thorsson, V., Ranish, J. A., Christmas, R., Buhler, J., Eng, J. K., Bumgarner, R., Goodlett, D. R., Aebersold, R., & Hood, L. (2001) · Science · doi:10.1126/science.292.5518.929 · Cited by 2,050 · see citing works
Genetic network inference: from co-expression clustering to reverse engineering
D’haeseleer, P., Liang, S., & Somogyi, R. (2000) · Bioinformatics · doi:10.1093/bioinformatics/16.8.707 · Cited by 943 · see citing works
Parameter estimation in biochemical pathways: a comparison of global optimization methods
Moles, C. G., Mendes, P., & Banga, J. R. (2003) · Genome Research · doi:10.1101/gr.1262503 · Cited by 858 · see citing works
Parametric Sensitivity in Chemical Systems
Varma, A., Morbidelli, M., & Wu, H. (1999) · Cambridge University Press eBooks · doi:10.1017/cbo9780511721779 · Cited by 472 · see citing works
Reverse engineering gene networks: integrating genetic perturbations with dynamical modeling
Tegnér, J., Yeung, M. K. S., Hasty, J., & Collins, J. J. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0933416100 · Cited by 433 · see citing works
Discrimination Among Mechanistic Models
Box, G. E. P., & Hill, W. J. (1967) · Technometrics · doi:10.1080/00401706.1967.10490441 · Cited by 422 · see citing works
Swameye, I., Müller, T. G., Timmer, J., Sandra, O., & Klingmüller, U. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0237333100 · Cited by 316 · see citing works
A test case of correlation metric construction of a reaction pathway from measurements
Arkin, A., Shen, P., & Ross, J. (1997) · Science · doi:10.1126/science.277.5330.1275 · Cited by 291 · see citing works
Statistical tools for optimal dynamic model building
Asprey, S. P., & Macchietto, S. (2000) · Computers & Chemical Engineering · doi:10.1016/s0098-1354(00)00328-8 · Cited by 244 · see citing works
Zak, D. E., Gonye, G. E., Schwaber, J. S., & Doyle, F. J. (2003) · Genome Research · doi:10.1101/gr.1198103 · Cited by 177 · see citing works
Self-citationStatistical construction of chemical reaction mechanisms from measured time-series
Arkin, A., & Ross, J. (1995) · The Journal of Physical Chemistry · doi:10.1021/j100003a020 · Cited by 143 · see citing works
Simulation methods for optimal experimental design in systems biology
Faller, D., Klingmüller, U., & Timmer, J. (2003) · SIMULATION · doi:10.1177/0037549703040937 · Cited by 137 · see citing works
REVERSE ENGINEERING OF METABOLIC PATHWAYS FROM OBSERVED DATA USING GENETIC PROGRAMMING
Koza, J. R., Mydlowec, W., Lanza, G., Yu, J., & Keane, M. A. (2000) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789814447362_0043 · Cited by 114 · see citing works
The Organization of Metabolic Reaction Networks
Kremling, A., Bettenbrock, K., Laube, B., Jahreis, K., Lengeler, J. W., & Gilles, E. D. (2001) · Metabolic Engineering · doi:10.1006/mben.2001.0199 · Cited by 97 · see citing works
Self-citationPerspectives on systems biology
Kitano, H. (2000) · New Generation Computing · doi:10.1007/bf03037529 · Cited by 95 · see citing works
Modular modeling of cellular systems with ProMoT/Diva
Ginkel, M., Kremling, A., Nutsch, T., Rehner, R., & Gilles, E. D. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btg128 · Cited by 95 · see citing works
Self-citationSequential experimental design for model discrimination in the case of multiple responses
Ferraris, G. B., Forzatti, P., Emig, G., et al. (1984) · Chemical Engineering Science · doi:10.1016/0009-2509(84)80132-3 · Cited by 78 · see citing works
Chen, B. H., & Asprey, S. P. (2003) · Industrial & Engineering Chemistry Research · doi:10.1021/ie0203025 · Cited by 72 · see citing works
Banga, J. R., Versyck, K. J. E., & van Impe, J. F. M. (2002) · Industrial & Engineering Chemistry Research · doi:10.1021/ie010183d · Cited by 69 · see citing works
Repsilber, D., Liljenström, H., & Andersson, S. G. E. (2002) · Biosystems · doi:10.1016/s0303-2647(02)00019-9 · Cited by 52 · see citing works
Optimal dynamic experiments for bioreactor model discrimination
Cooney, M. J., & McDonald, K. A. (1995) · Applied Microbiology and Biotechnology · doi:10.1007/bf02431915 · Cited by 33 · see citing works
On-Line Application of Parameter Estimation Accuracy to Biotechnical Processes
Posten, C. H., & Munack, A. (1990) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc.1990.4791117 · Cited by 22 · see citing works
Some Improvements in the Identification of Bioprocesses
Munack, A. (1992) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)50331-6 · Cited by 14 · see citing works
Towards a Virtual Biological Laboratory
Stelling, J., Kremling, A., Ginkel, M., Bettenbrock, K., & Gilles, E. D. (2001) · The MIT Press eBooks · doi:10.7551/mitpress/3087.003.0015 · Cited by 14 · see citing works
Self-citationAn optimized step-size random search (OSSRS)
Sheela, B. V. (1979) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/0045-7825(79)90035-5 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Towards a rigorous assessment of systems biology models: the DREAM3 challenges
Prill, R. J., Marbach, D., Saez-Rodriguez, J., Sorger, P. K., Alexopoulos, L. G., Xue, X., Clarke, N. D., Altan-Bonnet, G., & Stolovitzky, G. (2010) · PLoS ONE · doi:10.1371/journal.pone.0009202 · Cited by 439 · see citing works
A hybrid approach for efficient and robust parameter estimation in biochemical pathways
Rodriguez-Fernandez, M., Mendes, P., & Banga, J. R. (2006) · Biosystems · doi:10.1016/j.biosystems.2005.06.016 · Cited by 304 · see citing works
Novel metaheuristic for parameter estimation in nonlinear dynamic biological systems
Rodriguez-Fernandez, M., Egea, J. A., & Banga, J. R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-483 · Cited by 278 · see citing works
Systems biology: experimental design
Kreutz, C., & Timmer, J. (2009) · The FEBS Journal · doi:10.1111/j.1742-4658.2008.06843.x · Cited by 257 · see citing works
Reverse engineering and identification in systems biology: strategies, perspectives and challenges
Villaverde, A. F., & Banga, J. R. (2014) · Journal of The Royal Society Interface · doi:10.1098/rsif.2013.0505 · Cited by 229 · see citing works
An iterative identification procedure for dynamic modeling of biochemical networks
Balsa-Canto, E., Alonso, A. A., & Banga, J. R. (2010) · BMC Systems Biology · doi:10.1186/1752-0509-4-11 · Cited by 202 · see citing works
Sloppy models, parameter uncertainty, and the role of experimental design
Apgar, J. F., Witmer, D. K., White, F. M., & Tidor, B. (2010) · Molecular BioSystems · doi:10.1039/b918098b · Cited by 146 · see citing works
Parameter estimation and optimal experimental design
Banga, J. R., & Balsa-Canto, E. (2008) · Essays in Biochemistry · doi:10.1042/bse0450195 · Cited by 141 · see citing works
Computational procedures for optimal experimental design in biological systems
Balsa-Canto, E., Alonso, A. A., & Banga, J. R. (2008) · IET Systems Biology · doi:10.1049/iet-syb:20070069 · Cited by 124 · see citing works
Inverse problems in systems biology
Engl, H. W., Flamm, C., Kügler, P., Lu, J., Müller, S., & Schuster, P. (2009) · Inverse Problems · doi:10.1088/0266-5611/25/12/123014 · Cited by 122 · see citing works
Stimulus Design for Model Selection and Validation in Cell Signaling
Apgar, J. F., Toettcher, J. E., Endy, D., White, F. M., & Tidor, B. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.0040030 · Cited by 79 · see citing works
Systems biology—an engineering perspective
Kremling, A., & Saez-Rodriguez, J. (2007) · Journal of Biotechnology · doi:10.1016/j.jbiotec.2007.02.009 · Cited by 76 · see citing works
Self-citationMélykúti, B., August, E., Papachristodoulou, A., & El-Samad, H. (2010) · BMC Systems Biology · doi:10.1186/1752-0509-4-38 · Cited by 58 · see citing works
An optimal experimental design approach to model discrimination in dynamic biochemical systems
Skanda, D., & Lebiedz, D. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq074 · Cited by 42 · see citing works
Optimal design of stimulus experiments for robust discrimination of biochemical reaction networks
Flassig, R. J., & Sundmacher, K. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts585 · Cited by 31 · see citing works
A global parallel model based design of experiments method to minimize model output uncertainty
Bazil, J. N., Buzzard, G. T., & Rundell, A. E. (2012) · Bulletin of Mathematical Biology · doi:10.1007/s11538-011-9686-9 · Cited by 28 · see citing works
Robust optimal design of experiments for model discrimination using an interactive software tool
Stegmaier, J., Skanda, D., & Lebiedz, D. (2013) · PLoS ONE · doi:10.1371/journal.pone.0055723 · Cited by 15 · see citing works
- Authors
- Kremling, A., Fischer, S., Gadkar, K., Doyle, F. J., Sauter, T., Bullinger, E., Allgöwer, F., & Gilles, E. D.
- Affiliations
- Max Planck Institute for Dynamics of Complex Technical Systems (DE)
- University of California, Santa Barbara (US)
- University of Stuttgart (DE)
- Corresponding author
- Kremling, Andreas
- Venue
- Genome Research
- Publisher
- Cold Spring Harbor Laboratory
- Published
- 2004-09
- Volume
- 14
- Issue
- 9
- Pages
- 1773-1785
- ISSN
- 1088-9051 (ISSN-L)
- PMC
- PMC515324
- OpenAlex
- W2156154713
- Cited by
- 127
- References
- 28
- Citations by year
- 2025: 1 · 2024: 1 · 2021: 1 · 2020: 1 · 2019: 1 · 2018: 4 · 2017: 4 · 2016: 5 · 2015: 6 · 2014: 9 · 2013: 11 · 2012: 7
- Type
- posted content
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Genome Research (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Max Planck Institute for Plasma Physics (submittedVersion)
- ORBi (University of Liège) (submittedVersion)
- Open Repository and Bibliography (University of Luxembourg) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Funders
- National Science Foundation (0000961)
- Alexander von Humboldt-Stiftung
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Microbial Metabolic Engineering and Bioproduction
- Physical Sciences › Engineering › Control and Systems Engineering › Control Systems and Identification
- Keywords
- reverse engineering, model discrimination, parameter identifiability, parameter estimation, experimental design, bioreactors, fed-batch fermentation, benchmark problems, nonlinear models, linear models
- MeSH terms
- Algorithms, Biochemistry, Biomedical Engineering, Computer Simulation, Genes, Genetic Engineering, Kinetics, Models, Genetic, Research Design, Software, Nonlinear Dynamics, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
