Tellurium notebooks—An environment for reproducible dynamical modeling in systems biology
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
The considerable difficulty encountered in reproducing the results of published dynamical models limits validation, exploration and reuse of this increasingly large biomedical research resource. To address this problem, we have developed Tellurium Notebook, a software system that facilitates building reproducible dynamical models and reusing models by 1) supporting the COMBINE archive format during model development for capturing model information in an exchangeable format and 2) enabling users to easily simulate and edit public COMBINE-compliant models from public repositories to facilitate studying model dynamics, variants and test cases. Tellurium Notebook, a Python–based Jupyter–like environment, is designed to seamlessly inter-operate with these community standards by automating conversion between COMBINE standards formulations and corresponding in–line, human–readable representations. Thus, Tellurium brings to systems biology the strategy used by other literate notebook systems such as Mathematica. These capabilities allow users to edit every aspect of the standards–compliant models and simulations, run the simulations in–line, and re–export to standard formats. We provide several use cases illustrating the advantages of our approach and how it allows development and reuse of models without requiring technical knowledge of standards. Adoption of Tellurium should accelerate model development, reproducibility and reuse. Author summary There is considerable value to systems and synthetic biology in creating reproducible models. An essential element of reproducibility is the use of community standards, an often challenging undertaking for modelers. This article describes Tellurium Notebook, a tool for developing dynamical models that provides an intuitive approach to building and reusing models built with community standards. Tellurium automates embedding human–readable representations of COMBINE archives in literate coding notebooks, bringing to systems biology this strategy central to other literate notebook systems such as Mathematica. We show that the ability to easily edit this human–readable representation enables users to test models under a variety of conditions, thereby providing a way to create, reuse, and modify standard–encoded models and simulations, regardless of the user’s level of technical knowledge of said standards.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hucka, M., Finney, A., Sauro, H. M., Bolouri, H., Doyle, J. C., Kitano, H., Arkin, A. P., Bornstein, B. J., Bray, D., Cornish-Bowden, A., et al. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btg015 · Cited by 3,060
Self-citationCOPASI—a COmplex PAthway SImulator
Hoops, S., Sahle, S., Gauges, R., Lee, C., Pahle, J., Simus, N., Singhal, M., Xu, L., Mendes, P., & Kummer, U. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl485 · Cited by 2,618
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449
COBRApy: COnstraints-Based Reconstruction and Analysis for Python
Ebrahim, A., Lerman, J. A., Palsson, B. O., & Hyduke, D. R. (2013) · BMC Systems Biology · doi:10.1186/1752-0509-7-74 · Cited by 1,378
A Whole-Cell Computational Model Predicts Phenotype from Genotype
Karr, J. R., Sanghvi, J. C., Macklin, D. N., Gutschow, M. V., Jacobs, J. M., Bolival, B., Assad-Garcia, N., Glass, J. I., & Covert, M. W. (2012) · Cell · doi:10.1016/j.cell.2012.05.044 · Cited by 1,300 · cited 1×
“Examples include the Mycoplasma genitalium whole–cell model [1] and the central metabolism of E. coli [2].”
BackgroundTen Simple Rules for Reproducible Computational Research
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003285 · Cited by 880
Le Novere, N. (2006) · Nucleic Acids Research · doi:10.1093/nar/gkj092 · Cited by 762
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369
An Overview of CellML 1.1, a Biological Model Description Language
Cuellar, A. A., Lloyd, C. M., Nielsen, P. F., Bullivant, D. P., Nickerson, D. P., & Hunter, P. J. (2003) · SIMULATION · doi:10.1177/0037549703040939 · Cited by 345
Web-based kinetic modelling using JWS Online
Olivier, B. G., & Snoep, J. L. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth200 · Cited by 326
Controlled vocabularies and semantics in systems biology
Courtot, M., Juty, N., Knüpfer, C., Waltemath, D., Zhukova, A., Dräger, A., Dumontier, M., Finney, A., Golebiewski, M., Hastings, J., et al. (2011) · Molecular Systems Biology · doi:10.1038/msb.2011.77 · Cited by 316 · cited 2×
“SED–ML allows specifying the exact numerical algorithms needed to run a simulation using the Kinetic Simulation Algorithm Ontology (KiSAO) [20], which includes widely used ODE (e.g., LSODA [21, 22], CVODE [23]) and stochastic solvers (e.g., Gillespie direct method [24], Gibson algorithm [25]).”
MethodVirtual Cell modelling and simulation software environment
Moraru, I. I., Schaff, J. C., Slepchenko, B. M., Blinov, M. L., Morgan, F., Lakshminarayana, A., Gao, F., Li, Y., & Loew, L. M. (2008) · IET Systems Biology · doi:10.1049/iet-syb:20080102 · Cited by 236
COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project
Bergmann, F. T., Adams, R., Moodie, S., Cooper, J., Glont, M., Golebiewski, M., Hucka, M., Laibe, C., Miller, A. K., Nickerson, D. P., et al. (2014) · BMC Bioinformatics · doi:10.1186/s12859-014-0369-z · Cited by 173
Self-citationABC-SysBio approximate Bayesian computation in Python with GPU support
Liepe, J., Barnes, C., Cule, E., Erguler, K., Kirk, P., Toni, T., & Stumpf, M. P. H. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq278 · Cited by 136
SBML Level 3 Package: Flux Balance Constraints version 2
Olivier, B. G., & Bergmann, F. T. (2018) · Journal of Integrative Bioinformatics · doi:10.1515/jib-2017-0082 · Cited by 94
Waltemath, D., & Wolkenhauer, O. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2555481 · Cited by 49
Where next for the reproducibility agenda in computational biology?
Lewis, J., Breeze, C. E., Charlesworth, J., Maclaren, O. J., & Cooper, J. (2016) · BMC Systems Biology · doi:10.1186/s12918-016-0288-x · Cited by 23
Juty, N., & le Novère, N. (2013) · Encyclopedia of Systems Biology · doi:10.1007/978-1-4419-9863-7_1287 · Cited by 16
Cited by
BioModels—15 years of sharing computational models in life science
Malik-Sheriff, R. S., Glont, M., Nguyen, T. V. N., Tiwari, K., Roberts, M. G., Xavier, A., Vu, M. T., Men, J., Maire, M., Kananathan, S., et al. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz1055 · Cited by 361
SBML Level 3: an extensible format for the exchange and reuse of biological models
Keating, S. M., Waltemath, D., König, M., Zhang, F., Dräger, A., Chaouiya, C., Bergmann, F. T., Finney, A., Gillespie, C. S., Helikar, T., et al. (2020) · Molecular Systems Biology · doi:10.15252/msb.20199110 · Cited by 285
Self-citationTowards reproducible computational drug discovery
Schaduangrat, N., Lampa, S., Simeon, S., Gleeson, M. P., Spjuth, O., & Nantasenamat, C. (2020) · Journal of Cheminformatics · doi:10.1186/s13321-020-0408-x · Cited by 208
Catalyst: Fast and flexible modeling of reaction networks
Loman, T. E., Ma, Y., Ilin, V., Gowda, S., Korsbo, N., Yewale, N., Rackauckas, C., & Isaacson, S. A. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011530 · Cited by 47
Recent advances in biomedical simulations: a manifesto for model engineering
Hellerstein, J. L., Gu, S., Choi, K., & Sauro, H. M. (2019) · F1000Research · doi:10.12688/f1000research.15997.1 · Cited by 20
Self-citation
Metadata
- Authors
- Medley, J. K., Choi, K., König, M., Smith, L., Gu, S., Hellerstein, J., Sealfon, S. C., & Sauro, H. M.
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-06-15
- Volume
- 14
- Issue
- 6
- Pages
- e1006220
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W2807821194
- Cited by
- 55
- References
- 19
- FWCI
- 10.91
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.