The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; none of the 14 sources its claims cite is checked for support yet.
Sloppy models, parameter uncertainty, and the role of experimental design
Open accessAbstract
Computational models are increasingly used to understand and predict complex biological phenomena. These models contain many unknown parameters, at least some of which are difficult to measure directly, and instead are estimated by fitting to time-course data. Previous work has suggested that even with precise data sets, many parameters are unknowable by trajectory measurements. We examined this question in the context of a pathway model of epidermal growth factor (EGF) and neuronal growth factor (NGF) signaling. Computationally, we examined a palette of experimental perturbations that included different doses of EGF and NGF as well as single and multiple gene knockdowns and overexpressions. While no single experiment could accurately estimate all of the parameters, experimental design methodology identified a set of five complementary experiments that could. These results suggest optimism for the prospects for calibrating even large models, that the success of parameter estimation is intimately linked to the experimental perturbations used, and that experimental design methodology is important for parameter fitting of biological models and likely for the accuracy that can be expected from them.
Overview
CoChat Summary
TL;DR
Using a 48-parameter EGF/NGF signaling model, this study shows that parameter uncertainty is not inherent: a greedy experimental-design search found five complementary perturbation experiments that determine all 48 rate parameters to within 10%. Arbitrary or individually best experiments were much less effective, indicating that complementarity of experimental perturbations is critical for model calibration.
Key findings
- Five complementary experiments (Table I) determined all 48 rate parameters to within 10%; the best individual experiment defined only 29 of 48 parameter directions to that accuracy.
- Random or “best singles” selection could not determine all 48 parameters with up to 20 experiments, whereas the greedy experimental-design procedure required only five.
- Simpler perturbation classes were less effective: EGF/NGF doses alone established 25 of 48 parameters (requiring 2 experiments, with no improvement from the remaining 23); single expression changes established 45 of 48 (17 experiments); double changes established 47 of 48 (8 experiments); triple changes established all 48 (5 experiments).
- Relaxing the error threshold to 37% allowed all but two parameters to be estimated with two experiments and all 48 with four experiments.
Methods
- Model: EGF/NGF signaling model from BioModels (BIOMD0000000033), consisting of 19 proteins, 32 species, 26 reactions, and 48 parameters (22 Michaelis constants, 22 catalytic rate constants, 2 second-order association constants, 2 first-order dissociation constants); initial concentrations assumed known.
- Experimental design: candidate palette combined EGF/NGF doses with up to three simultaneous protein overexpression or knockdown changes, yielding 164,500 computational experiments; a greedy search selected experiments by maximizing the number of parameter eigendirections below a threshold.
- Analysis: Hessians of the chi-squared fit metric were computed at the best-fit parameter set using MATLAB SimBiology v2.4, ODE15S, complex-step finite differencing, and trapezoidal integration; assumed 10% Gaussian relative measurement error and 100 time points per species, with λ_thresh = 0.005 for 10% relative error.
Limitations
- Stated: the Hessian was evaluated only at the optimal parameter set, assuming Gaussian log-parameter errors; this may not hold for poorly determined parameter directions, though the approximation improves as complementary experiments reduce uncertainty.
- Stated: predictions assume the model topology is sufficiently correct and that no bifurcations or other anomalies occur within the remaining parameter uncertainty; fitting used full trajectories of all concentrations, whereas current experimental technology generally probes only a subset of species and time points.
- Stated/evident: the greedy search does not necessarily produce the optimal subset of a given size, so the results are an upper bound on the number of experiments required; the study is in silico on one fixed EGF/NGF model with known initial concentrations and no gene-expression regulation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 1 Not valid · 9 Not checked · 72 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While no single experiment could accurately estimate all of the parameters, experimental design methodology identified a set of five complementary experiments that could.”
- Also stated in: § All parameters can be determined to high accuracy · plain
“It is striking to observe that, by properly choosing the correct combination of experiments, only five total experiments are sufficient to determine all 48 directions (and, indeed, all 48 actual parameters) to within 10%”
- Also stated in: § Discussion · hedged
“all 48 rate constant parameters could in principle be fit to within 10% of their nominal value using concentration time courses from five multi-perturbation experiments.”
§ Sloppy Models, Parameter Uncertainty, and the Role of Experimental Design > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § All parameters can be determined to high accuracy · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but doses of EGF/NGF alone were only able to establish just over half of the rate constants (25 of 48, requiring just 2 experiments with no improvement resulting from the remaining 23 experiments in the class).”
§ Sloppy Models, Parameter Uncertainty, and the Role of Experimental Design > Relative resolving power of different experiment types
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most rate constants could be determined using single changes in protein expression (45 of 48, but requiring 17 experiments)”
§ Sloppy Models, Parameter Uncertainty, and the Role of Experimental Design > Relative resolving power of different experiment types
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The best individual experiment in this set defined only 29 of the 48 rate parameter directions to this high level of accuracy.”
§ Sloppy Models, Parameter Uncertainty, and the Role of Experimental Design > All parameters can be determined to high accuracy
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Likewise, as the stringency was increased, greater experimental effort was required to establish a given number of parameters.”
§ Sloppy Models, Parameter Uncertainty, and the Role of Experimental Design > More stringent parameterization can be achieved with greater experimental effort
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
44 more claims
Create a free account to see all 82 claims and to verify the 14 references still open.
- It may state two separate claims as one. 27
- Probably not a substantial claim. 12
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This uncertainty, it has turned out, can play a pivotal role in the analysis of a system’s dynamics16–18.”
From A framework for parameter estimation and model selection from experimental data in systems biology using approximate Bayesian computation (2014) · cites this paper as [17]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It is known that the practical identifiability of a model can be improved by means of an adequate experimental design [2, 4, 41, 24], and a similar procedure can be used to reduce sloppiness.”
From On the relationship between sloppiness and identifiability (2016) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“The importance of the role played by experimental design in this task had been stressed e.g. in [2, 32].”
From On the relationship between sloppiness and identifiability (2016) · cites this paper as [2]
In the citing paper
“Practical identifiability problems can be surmounted by more informative data and, ideally, by OED (see related comments by Apgar et al. [208]).”
From Reverse engineering and identification in systems biology: strategies, perspectives and challenges (2014) · cites this paper as [208]
In the citing paper
“However, advances in such technologies, coupled with new ways of introducing perturbations and the use of OED methods, should lead to identifiable dynamic models [208,211], so we should be optimistic about the calibration of these models.”
From Reverse engineering and identification in systems biology: strategies, perspectives and challenges (2014) · cites this paper as [208]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Sloppy Models, Parameter Uncertainty, and the Role of Experimental Design > Results
“NGF (4.52×10 7 , 4.52×10 5 , 4.52×10 3 , 45.2, or 0 molecules/cell-volume)”
§ Sloppy Models, Parameter Uncertainty, and the Role of Experimental Design > Computing Hessians
“Table I Parameter-Defining Experimental Set Exp EGF (mol./cell)NGF (mol./cell)Overexpressed Knocked Down 1 1.00 × 10 5 4.56 × 10 7 Sos, Ras, C3G 2 1.00 × 10 1 4.56 × 10 1 Mek, Erk Raf1PPtase”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Sloppy Models, Parameter Uncertainty, and the Role of Experimental Design > Introduction
“Recently Gutenkunst et al.21 examined parameter uncertainty for 17 models in the EMBL BioModels Database.22”
§ Sloppy Models, Parameter Uncertainty, and the Role of Experimental Design > References
“21. Haunschild MD, Freisleben B, Takors R, Wiechert W. Bioinformatics. 2005; 21:1617-1625. [PubMed: 15604106] 22. Gutenkunst RN, Waterfall JJ, Casey FP, Brown KS, Myers CR, Sethna JP. PLoS Comput. Biol. 2007; 3:1871-1878. [PubMed: 17922568]”
Citations
Cites
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
Untangling the ErbB signalling network
Yarden, Y., & Sliwkowski, M. X. (2001) · Nature Reviews Molecular Cell Biology · doi:10.1038/35052073 · Cited by 6,974
Systems biology: a brief overview
Kitano, H. (2002) · Science · doi:10.1126/science.1069492 · Cited by 4,265
Shampine, L. F., & Reichelt, M. W. (1997) · SIAM Journal on Scientific Computing · doi:10.1137/s1064827594276424 · Cited by 3,599
HIV-1 Dynamics in Vivo: Virion Clearance Rate, Infected Cell Life-Span, and Viral Generation Time
Perelson, A. S., Neumann, A. U., Markowitz, M., Leonard, J. M., & Ho, D. D. (1996) · Science · doi:10.1126/science.271.5255.1582 · Cited by 3,531
A gene atlas of the mouse and human protein-encoding transcriptomes
Su, A. I., Wiltshire, T., Batalov, S., Lapp, H., Ching, K. A., Block, D., Zhang, J., Soden, R., Hayakawa, M., Kreiman, G., et al. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0400782101 · Cited by 3,475
Nonlinear Parameter Estimation
Brubaker, T. A., & O'Keefe, K. R. (1979) · Analytical Chemistry · doi:10.1021/ac50049a796 · Cited by 2,134
The EGFR family and its ligands in human cancer
Yarden, Y. (2001) · European Journal of Cancer · doi:10.1016/s0959-8049(01)00230-1 · Cited by 1,587
Universally Sloppy Parameter Sensitivities in Systems Biology Models
Gutenkunst, R. N., Waterfall, J. J., Casey, F. P., Brown, K. S., Myers, C. R., & Sethna, J. P. (2007) · PLoS Computational Biology · doi:10.1371/journal.pcbi.0030189 · Cited by 1,261
Ultrasensitivity in the mitogen-activated protein kinase cascade.
Huang, C. Y., & Ferrell, J. E. (1996) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.93.19.10078 · Cited by 1,186
Research works printed but not linked: 11
- Akaike, 1973, Second International Symposium on Information Theory, pp. 267–281
- Schwartz, 1978, Ann. Stat., 6, 461
- Faller, 2003, Simul.–Trans. Soc. Model. Simul. Int., 79, 717
- Kutalik, 2004, BioSystems, 75, 43, 10.1016/j.biosystems.2004.03.007 · DOI · Paper page
- Donckels, 2009, J. Comput. Chem., 30, 2064, 10.1002/jcc.21171 · DOI
- Cho, 2003, Simul.–Trans. Soc. Model. Simul. Int., 79, 726
- Casey, 2007, IET Syst. Biol., 1, 190, 10.1049/iet-syb:20060065 · DOI · Paper page
- Jaqaman, 2006, Nat. Rev. Mol. Cell Biol., 7, 813, 10.1038/nrm2030 · DOI · Paper page
- Britsch, 2007, Advances in Anatomy Embryology and Cell Biology, 190, 1
- Seber, 2003, Nonlinear Regression
- Bertsekas, 2008, Introduction to Probability
Cited by
Perspective: Sloppiness and emergent theories in physics, biology, and beyond
Transtrum, M. K., Machta, B. B., Brown, K. S., Daniels, B. C., Myers, C. R., & Sethna, J. P. (2015) · The Journal of Chemical Physics · doi:10.1063/1.4923066 · Cited by 311
Parameter Space Compression Underlies Emergent Theories and Predictive Models
Machta, B. B., Chachra, R., Transtrum, M. K., & Sethna, J. P. (2013) · Science · doi:10.1126/science.1238723 · Cited by 285
Reverse engineering and identification in systems biology: strategies, perspectives and challenges
Villaverde, A. F., & Banga, J. R. (2014) · Journal of The Royal Society Interface · doi:10.1098/rsif.2013.0505 · Cited by 229
Efficient search, mapping, and optimization of multi-protein genetic systems in diverse bacteria
Farasat, I., Kushwaha, M., Collens, J., Easterbrook, M., Guido, M., & Salis, H. M. (2014) · Molecular Systems Biology · doi:10.15252/msb.20134955 · Cited by 235
Liepe, J., Kirk, P., Filippi, S., Toni, T., Barnes, C. P., & Stumpf, M. P. H. (2014) · Nature Protocols · doi:10.1038/nprot.2014.025 · Cited by 217
Metadata
- Authors
- Apgar, J. F., Witmer, D. K., White, F. M., & Tidor, B.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Koch Institute for Integrative Cancer Research At MIT (US)
- Corresponding author
- Tidor, Bruce
- Venue
- Molecular BioSystems
- Publisher
- Oxford University Press (OUP)
- Published
- 2010-06-17
- Volume
- 6
- Issue
- 10
- Pages
- 1890-1900
- ISSN
- 1742-2051 (ISSN-L)
- 1742-206X
- PMC
- PMC3505121
- OpenAlex
- W2116361328
- Semantic Scholar
- 6d2587e756e59a2edf3bc23628241e2ea308d755
- Cited by
- 146
- References
- 48
- Citations by year
- 2026: 7 · 2025: 7 · 2024: 6 · 2023: 7 · 2022: 7 · 2021: 8 · 2020: 8 · 2019: 11 · 2018: 10 · 2017: 9 · 2016: 7 · 2015: 12 · 2014: 20 · 2013: 9 · 2012: 8
- FWCI
- 3.49
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Molecular BioSystems (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
- PMC
- Funders
- MIT Portugal
- National Institutes of Health (CA112967)
- National Cancer Institute (U54CA112967)
- National Institute of General Medical Sciences (R01GM065418)
- National Institute of Environmental Health Sciences (P30ES002109)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Physical Sciences › Engineering › Control and Systems Engineering › Control Systems and Identification
- Keywords
- parameter uncertainty, experimental design, parameter estimation, EGF signaling, gene knockdown, gene overexpression, time course data, model calibration
- MeSH terms
- Epidermal Growth Factor, Models, Theoretical, Research Design, Signal Transduction, Nerve Growth Factor, Uncertainty, Gene Knockdown Techniques
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Sloppy models, parameter uncertainty, and the role of experimental design. https://app.cochat.ai/papers/10.1039/b918098b
- The paper
- Apgar, J. F., Witmer, D. K., White, F. M., & Tidor, B. (2010). Sloppy models, parameter uncertainty, and the role of experimental design. Molecular BioSystems. https://doi.org/10.1039/b918098b
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.