Standards, dissemination, and best practices in systems biology
Open accessAbstract
The reproducibility of scientific research is crucial to the success of the scientific method. Here, we review the current best practices when publishing mechanistic models in systems biology. We recommend, where possible, to use software engineering strategies such as testing, verification, validation, documentation, versioning, iterative development, and continuous integration. In addition, adhering to the Findable, Accessible, Interoperable, and Reusable modeling principles allows other scientists to collaborate and build off of each other's work. Existing standards such as Systems Biology Markup Language, CellML, or Simulation Experiment Description Markup Language can greatly improve the likelihood that a published model is reproducible, especially if such models are deposited in well-established model repositories. Where models are published in executable programming languages, the source code and their data should be published as open-source in public code repositories together with any documentation and testing code. For complex models, we recommend container-based solutions where any software dependencies and the run-time context can be easily replicated.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reproducibility is directly connected to credibility, but even reproducibility remains a challenge.
“Reproducibility, the ability to recreate a model and data de novo and obtain the same result [5], is directly connected to credibility, but even reproducibility remains a challenge.”
From Adapting modeling and simulation credibility standards to computational systems biology (2023) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hodgkin, A. L., & Huxley, A. F. (1952) · The Journal of Physiology · doi:10.1113/jphysiol.1952.sp004764 · Cited by 23,191 · see citing works
Hucka, M., Finney, A., Sauro, H. M., Bolouri, H., Doyle, J. C., Kitano, H., Arkin, A. P., Bornstein, B. J., Bray, D., Cornish-Bowden, A., et al. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btg015 · Cited by 3,060 · see citing works
Self-citationWhat does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · see citing works
Social coding in GitHub: transparency and collaboration in an open software repository
Dabbish, L., Stuart, C., Tsay, J., & Herbsleb, J. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145396 · Cited by 999 · see citing works
The systems biology graphical notation.
Novère, N. L., Hucka, M., Mi, H., Moodie, S., Schreiber, F., Sorokin, A., Demir, E., Wegner, K., Aladjem, M. I., Wimalaratne, S. M., et al. (2009) · Nature Biotechnology · doi:10.1038/nbt.1558 · Cited by 903 · see citing works
Self-citationHerrgård, M. J., Swainston, N., Dobson, P., Dunn, W. B., Arga, K. Y., Arvas, M., Blüthgen, N., Borger, S., Costenoble, R., Heinemann, M., et al. (2008) · Nature Biotechnology · doi:10.1038/nbt1492 · Cited by 588 · see citing works
FAIR Principles: Interpretations and Implementation Considerations
Jacobsen, A., Azevedo, R. D. M., Juty, N., Batista, D., Coles, S. J., Ronald Cornet, A., Courtot, M., Crosas, M., Dumontier, M., Evelo, C. T., et al. (2019) · Data Intelligence · doi:10.1162/dint_r_00024 · Cited by 475 · see citing works
Towards FAIR Principles for Research Software
Lamprecht, A.-L., Garcia, L., Kuzak, M., Martinez, C., Arcila, R., Martin Del Pico, E., Dominguez Del Angel, V., van de Sandt, S., Ison, J., Martinez, P. A., et al. (2020) · Data Science · doi:10.3233/ds-190026 · Cited by 322 · see citing works
Zhu, X., Jee, G., Baker, N. R., de Sturler, E., Ort, D. R., & Long, S. P. (2005) · Planta · doi:10.1007/s00425-005-0064-4 · Cited by 274 · see citing works
SBML Level 3: an extensible format for the exchange and reuse of biological models
Keating, S. M., Waltemath, D., König, M., Zhang, F.-K., Dräger, A., Chaouiya, C., Bergmann, F. T., Finney, A., Gillespie, C. S., Helikar, T., et al. (2020) · Molecular Systems Biology · doi:10.15252/msb.20199110 · Cited by 270 · see citing works
Self-citationWaltemath, D., Adams, R., Bergmann, F. T., Hucka, M., Kolpakov, F., Miller, A. K., Moraru, I., Nickerson, D., Sahle, S., Snoep, J., et al. (2011) · BMC Systems Biology · doi:10.1186/1752-0509-5-198 · Cited by 269 · see citing works
Reproducibility standards for machine learning in the life sciences
Heil, B. J., Hoffman, M. M., Markowetz, F. M., Lee, S., Greene, C. S., & Hicks, S. C. (2021) · Nature Methods · doi:10.1038/s41592-021-01256-7 · Cited by 261 · see citing works
Hucka, M., Bergmann, F. T., Chaouiya, C., Dräger, A., Hoops, S., Keating, S. M., König, M., Novère, N. L., Myers, C. J., Olivier, B. G., et al. (2019) · Journal of Integrative Bioinformatics · doi:10.1515/jib-2019-0021 · Cited by 258 · see citing works
BiGG Models 2020: multi-strain genome-scale models and expansion across the phylogenetic tree
Norsigian, C. J., Pusarla, N., McConn, J. L., Yurkovich, J. T., Dräger, A., Palsson, B. Ø., & King, Z. A. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz1054 · Cited by 250 · see citing works
Twenty years of ModelDB and beyond: building essential modeling tools for the future of neuroscience
McDougal, R. A., Morse, T. M., Carnevale, T., Marenco, L. N., Wang, R., Migliore, M., Miller, P. L., Shepherd, G. M., & Hines, M. L. (2016) · Journal of Computational Neuroscience · doi:10.1007/s10827-016-0623-7 · Cited by 214 · see citing works
SABIO-RK: an updated resource for manually curated biochemical reaction kinetics
Wittig, U., Rey, M., Weidemann, A., Kania, R., & Müller, W. (2018) · Nucleic Acids Research · doi:10.1093/nar/gkx1065 · Cited by 204 · see citing works
COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project
Bergmann, F. T., Adams, R., Moodie, S., Cooper, J., Glont, M., Golebiewski, M., Hucka, M., Laibe, C., Miller, A. K., Nickerson, D. P., et al. (2014) · BMC Bioinformatics · doi:10.1186/s12859-014-0369-z · Cited by 173 · see citing works
Self-citationThe Physiome Model Repository 2
Yu, T., Lloyd, C. M., Nickerson, D. P., Cooling, M. T., Miller, A. K., Garny, A., Terkildsen, J. R., Lawson, J., Britten, R. D., Hunter, P. J., et al. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btq723 · Cited by 169 · see citing works
Making Models Match: Replicating an Agent-Based Model
Wilensky, U. J., & Rand, W. M. (2007) · RePEc: Research Papers in Economics · Cited by 168 · see citing works
Toward the automated generation of genome-scale metabolic networks in the SEED
DeJongh, M., Formsma, K., Boillot, P., Gould, J., Rycenga, M., & Best, A. A. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-139 · Cited by 144 · see citing works
Yeast 5 – an expanded reconstruction of the Saccharomyces cerevisiae metabolic network
Heavner, B., B, K. S. A., Barker, B., Mendes, P., & Walker, L. P. (2012) · BMC Systems Biology · doi:10.1186/1752-0509-6-55 · Cited by 142 · see citing works
Antimony: a modular model definition language
Smith, L. P., Bergmann, F., Chandran, D., & Sauro, H. M. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp401 · Cited by 128 · see citing works
Self-citationSEEK: a systems biology data and model management platform
Wolstencroft, K., Owen, S., Krebs, O., Nguyen, Q., Stanford, N., Golebiewski, M., Weidemann, A., Bittkowski, M., An, L., Shockley, D., et al. (2015) · BMC Systems Biology · doi:10.1186/s12918-015-0174-y · Cited by 113 · see citing works
Further developments towards a genome-scale metabolic model of yeast
Dobson, P. D., B, K. S. A., Jameson, D., Simeonidis, E., Lanthaler, K., Pir, P., Lü, C., Swainston, N., Dunn, W. B., Fisher, P., et al. (2010) · BMC Systems Biology · doi:10.1186/1752-0509-4-145 · Cited by 110 · see citing works
Tanaka, K., Henry, C. S., Zinner, J. F., Jolivet, E., Cohoon, M., Xia, F., Bidnenko, V., Ehrlich, S. D., Stevens, R. L., & Noirot, P. H. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks963 · Cited by 99 · see citing works
Emerging whole-cell modeling principles and methods
Goldberg, A. P., Szigeti, B., Yin, H.-C., Sekar, J. A. P., Roth, Y. D., & Karr, J. R. (2017) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2017.12.013 · Cited by 82 · see citing works
A quantitative model of nitrogen fixation in the presence of ammonium
Inomura, K., Bragg, J. G., Riemann, L., & Follows, M. J. (2018) · PLoS ONE · doi:10.1371/journal.pone.0208282 · Cited by 78 · see citing works
Neupane, J. B., Neupane, R. P., Luo, Y., Yoshida, W. Y., Sun, R., & Williams, P. G. (2019) · Organic Letters · doi:10.1021/acs.orglett.9b03216 · Cited by 70 · see citing works
SED-ML - An XML Format for the Implementation of the MIASE Guidelines
Köhn, D., & Novère, N. L. (2008) · Computational Methods in Systems Biology · doi:10.1007/978-3-540-88562-7_15 · Cited by 69 · see citing works
Singh, J. (2011) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.81919 · Cited by 64 · see citing works
Aller, S., Scott, A., Sarkar‐Tyson, M., & Soyer, O. S. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2018.0125 · Cited by 63 · see citing works
Towards an ecosystem model of infectious disease
Hassell, J. M., Newbold, T., Dobson, A. P., Linton, Y., Franklinos, L. H. V., Zimmerman, D. M., & Lohan, K. M. P. (2021) · Nature Ecology & Evolution · doi:10.1038/s41559-021-01454-8 · Cited by 61 · see citing works
Noll, P., & Henkel, M. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.10.018 · Cited by 60 · see citing works
Mulugeta, L., Drach, A., Erdemir, A., Hunt, C. A., Horner, M., Ku, J. P., Myers, J. G., Vadigepalli, R., & Lytton, W. W. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2018.00018 · Cited by 58 · see citing works
Do genome-scale models need exact solvers or clearer standards?
Ebrahim, A., Almaas, E., Bauer, E., Bordbar, A., Burgard, A. P., Chang, R. L., Dräger, A., Famili, I., Feist, A. M., Fleming, R. M., et al. (2015) · Molecular Systems Biology · doi:10.15252/msb.20156157 · Cited by 57 · see citing works
Bayesian inference of metabolic kinetics from genome-scale multiomics data
John, P. C. S., Strutz, J., Broadbelt, L. J., Tyo, K. E. J., & Bomble, Y. J. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007424 · Cited by 55 · see citing works
Community standards to facilitate development and address challenges in metabolic modeling
Carey, M. A., Dräger, A., Beber, M. E., Papin, J. A., & Yurkovich, J. T. (2020) · Molecular Systems Biology · doi:10.15252/msb.20199235 · Cited by 49 · see citing works
Best Practices for Making Reproducible Biochemical Models
Porubsky, V. L., Goldberg, A. P., Rampadarath, A. K., Nickerson, D. P., Karr, J. R., & Sauro, H. M. (2020) · Cell Systems · doi:10.1016/j.cels.2020.06.012 · Cited by 45 · see citing works
Self-citationPham, N., van Heck, R. G. A., van Dam, J. C. J., Schaap, P. J., Saccenti, E., & Diez, M. S. (2019) · Metabolites · doi:10.3390/metabo9020028 · Cited by 44 · see citing works
Reconstruction and analysis of the genome-scale metabolic model of Lactobacillus casei LC2W
Xu, N., Liu, J., Ai, L., & Liu, L. (2014) · Gene · doi:10.1016/j.gene.2014.10.034 · Cited by 37 · see citing works
The war over supercooled water
Smart, A. G. (2018) · Physics Today · doi:10.1063/pt.6.1.20180822a · Cited by 28 · see citing works
Sjögren, E., Bredberg, U., & Lennernäs, H. (2012) · Molecular Pharmaceutics · doi:10.1021/mp200218p · Cited by 26 · see citing works
Unit testing, model validation, and biological simulation
Sarma, G., Jacobs, T. W., Watts, M. D., Ghayoomie, S. V., Larson, S., & Gerkin, R. C. (2016) · F1000Research · doi:10.12688/f1000research.9315.1 · Cited by 26 · see citing works
Recent advances in biomedical simulations: a manifesto for model engineering
Hellerstein, J. L., Gu, S., Choi, K., & Sauro, H. M. (2019) · F1000Research · doi:10.12688/f1000research.15997.1 · Cited by 20 · see citing works
Self-citationWang, C., Li, S., Ademiloye, A. S., & Nithiarasu, P. (2021) · International Journal for Numerical Methods in Biomedical Engineering · doi:10.1002/cnm.3520 · Cited by 15 · see citing works
Publishing reproducible dynamic kinetic models
Porubsky, V. L., Smith, L. P., & Sauro, H. M. (2020) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa152 · Cited by 12 · see citing works
Self-citationExecutable Simulation Model of the Liver
König, M. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-801238-3.11682-4 · Cited by 10 · see citing works
libOmexMeta: enabling semantic annotation of models to support FAIR principles
Welsh, C., Nickerson, D. P., Rampadarath, A., Neal, M. L., Sauro, H. M., & Gennari, J. H. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab445 · Cited by 8 · see citing works
Self-citationLang, P. F., Chebaro, Y., Zheng, X., Sekar, J. A. P., Shaikh, B., Natale, D., & Karr, J. R. (2019) · Genome Biology · doi:10.1186/s13059-020-02025-z · Cited by 7 · see citing works
Dynamic Flux Balance Analysis Models in SBMLPreprint
König, M., Watanabe, L. H., Grzegorzewski, J., & Myers, C. (2018) · bioRxiv · doi:10.1101/245076 · Cited by 4 · see citing works
The Practice of Ensuring Repeatable and Reproducible Computational ModelsPreprint
Sauro, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05386 · Cited by 3 · see citing works
Self-citationReproducibility in Biomedical Research
(2019) · Elsevier eBooks · doi:10.1016/c2018-0-02296-3 · Cited by 3 · see citing works
Flagged first, then by citations
3 of the 55 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [16]Murray-Smith DJ, Testing and validation of computer simulation models, Cham: Springer. doi 10 (2015) 978– 3.
Cited by
Sorted by citations
Adapting modeling and simulation credibility standards to computational systems biology
Tatka, L. T., Smith, L. P., Hellerstein, J. L., & Sauro, H. M. (2023) · Journal of Translational Medicine · doi:10.1186/s12967-023-04290-5 · Cited by 25 · see citing works
Self-citationSystems-Level Modeling for CRISPR-Based Metabolic Engineering
Cardiff, R. A. L., Carothers, J. M., Zalatan, J. G., & Sauro, H. M. (2024) · ACS Synthetic Biology · doi:10.1021/acssynbio.4c00053 · Cited by 21 · see citing works
Self-citationneo4jsbml: import systems biology markup language data into the graph database Neo4j
Gricourt, G., Duigou, T., Dérozier, S., & Faulon, J. (2024) · PeerJ · doi:10.7717/peerj.16726 · Cited by 11 · see citing works
From Average Transient Transporter Currents to Microscopic Mechanism─A Bayesian Analysis
George, A., & Zuckerman, D. M. (2024) · The Journal of Physical Chemistry B · doi:10.1021/acs.jpcb.3c07025 · Cited by 10 · see citing works
From FAIR to CURE: guidelines for computational models of biological systems
Sauro, H. M., Agmon, E., Blinov, M., Gennari, J. H., Hellerstein, J. L., Heydarabadipour, A., Jardine, B. E., May, E., Nickerson, D., Smith, L. P., et al. (2026) · npj Systems Biology and Applications · doi:10.1038/s41540-026-00651-0 · Cited by 6 · see citing works
Self-citationLi, L. X., Aguilar, B. H., Gennari, J. H., & Qin, G. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06212-2 · Cited by 1 · see citing works
One Earth + One Health: An Agile, Evolutionary, System-of-Systems Convergence Paradigm
Little, J. C., Kaaronen, R. O., Muthukrishna, M., Elsawah, S., Bennett, M. S., Khayal, I. S., Hukkinen, J., Barton, C. M., Jakeman, A. J., & Farid, A. M. (2026) · Environmental Science & Technology · doi:10.1021/acs.est.5c11895 · Cited by 2 · see citing works
van Aalst, M., Lahlou, A., Hassan, T., Gaultier, W., Colliaux, D., & Matuszyńska, A. (2026) · PLoS Biology · doi:10.1371/journal.pbio.3003825 · Cited by 1 · see citing works
LM-Merger: A workflow for merging logical models with an application to gene regulationPreprint
Li, L. X., Aguilar, B. H., Gennari, J. H., & Qin, G. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.13.612961
FAIR principles for good modeling practice and standardsPreprint
Rosello, C., Hamilton, S. H., Grimm, V., Barton, M., Edmonds, B., Elsawah, S., Grant, W. E., Hengeveld, G., Jakeman, A. J., Koralewski, T. E., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6213393
In silico methodologies in computational medicine: From complex disease modeling to virtual clinics
Italia, M., Otero, J. G., Azzimonti, L., García‐Aznar, J. M., Dercole, F., & Belmonte-Beitia, J. (2026) · Biophysics Reviews · doi:10.1063/5.0285118
XML Languages and Simulators for Analyzing Molecular Communications Systems
Cevallos, Y., Naranjo, P. G. V., Inca, D., Pilligua, E., Tello-Oquendo, L., León, M., Gómez, C. V., Tene, T., Svozilík, J., Bodero-Poveda, E., et al. (2026) · Signals and communication technology · doi:10.1007/978-3-032-33040-6_2
Luthert, P. J., Faes, L., Foss, A. J. E., Fu, D. J., Hill, L. J., Huang, R., Kiel, C., Lee, W. H., Matei, Mirams, G. R., et al. (2026) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2026.101529
Zhang, X., Ji, C., & Hu, W. (2026) · Frontiers in Pharmacology · doi:10.3389/fphar.2026.1872382
Efficient computational frameworks for whole-cell models
Huggins, J. R., & Birtwistle, M. R. (2026) · Current Opinion in Systems Biology · doi:10.1016/j.coisb.2026.100610
- Authors
- Shin, J., Porubsky, V., Carothers, J., & Sauro, H. M.
- Affiliations
- University of Washington (US)
- Corresponding author
- Sauro, Herbert M
- Venue
- Current Opinion in Biotechnology
- Publisher
- Elsevier BV
- Published
- 2023-06
- Volume
- 81
- Pages
- 102922
- ISSN
- 0958-1669 (ISSN-L)
- PMC
- PMC10435326
- OpenAlex
- W4362450323
- Cited by
- 15
- References
- 55
- Citations by year
- 2026: 9 · 2025: 1 · 2024: 4 · 2023: 1
- FWCI
- 0.88
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- www.elsevier.com/open-access/userlicense/1.0/ (am, 304 d)
- doi.org/10.15223/policy-017 (stm-asf)
- doi.org/10.15223/policy-037 (stm-asf)
- doi.org/10.15223/policy-012 (stm-asf)
- doi.org/10.15223/policy-029 (stm-asf)
- doi.org/10.15223/policy-004 (stm-asf)
- Locations
- Current Opinion in Biotechnology (acceptedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion, other-oa)
- Funders
- National Science Foundation (P41GM109824, 1933453)
- U.S. Department of Energy (DEEE0008927)
- University of Washington (1933453, P41GM109824)
- National Institutes of Health (1933453)
- National Institute of General Medical Sciences (P41GM109824)
- National Institute of Biomedical Imaging and Bioengineering (U24EB028887, P41EB023912)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- systems biology, mechanistic model, model reproducibility, software engineering best practices, model verification and validation, model testing, SBML, CellML, model repository, open source software, containerization, continuous integration, model documentation
- MeSH terms
- Computer Simulation, Models, Biological, Programming Languages, Software, Reproducibility of Results, Systems Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
