Best Practices for Making Reproducible Biochemical Models
Open accessAbstract
Like many scientific disciplines, dynamical biochemical modeling is hindered by irreproducible results. This limits the utility of biochemical models by making them difficult to understand, trust, or reuse. We comprehensively list the best practices that biochemical modelers should follow to build reproducible biochemical model artifacts-all data, model descriptions, and custom software used by the model-that can be understood and reused. The best practices provide advice for all steps of a typical biochemical modeling workflow in which a modeler collects data; constructs, trains, simulates, and validates the model; uses the predictions of a model to advance knowledge; and publicly shares the model artifacts. The best practices emphasize the benefits obtained by using standard tools and formats and provides guidance to modelers who do not or cannot use standards in some stages of their modeling workflow. Adoption of these best practices will enhance the ability of researchers to reproduce, understand, and reuse biochemical models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Model reproducibility best practices describe dissemination as an essential part of reproducibility.
“Model reproducibility best practices describe dissemination as an essential part of reproducibility [52].”
From Adapting modeling and simulation credibility standards to computational systems biology (2023) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430 · see citing works
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423 · see citing works
The MIQE guidelines: minimum information for publication of quantitative real-time PCR experiments
Bustin, S. A., Benes, V., Garson, J. A., Hellemans, J., Huggett, J., Kubista, M., Mueller, R., Nolan, T., Pfaffl, M. W., Shipley, G. L., et al. (2009) · Clinical Chemistry · doi:10.1373/clinchem.2008.112797 · Cited by 16,303 · cited 1× · see citing works
“We encourage experimentalists to use appropriate ontologies and minimum information standards when recording experimental methods, conditions, and the historical record of the data (Bandrowski et al., 2016; Kazic, 2015; Orchard et al., 2007; Deutsch et al., 2008; Bustin et al., 2009; Brazma et al., 2001; Taylor et al.…”
MethodGolub, T. R., Slonim, D. K., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J. P., Coller, H., Loh, M. L., Downing, J. R., Caligiuri, M. A., et al. (1999) · Science · doi:10.1126/science.286.5439.531 · Cited by 11,516 · see citing works
Database resources of the National Center for Biotechnology Information: update
Wheeler, D. L. (2003) · Nucleic Acids Research · doi:10.1093/nar/gkh073 · Cited by 10,691 · see citing works
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2019) · Apollo - University of Cambridge Repository · doi:10.17863/cam.64134 · Cited by 10,460 · see citing works
Minimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109 · see citing works
Hucka, M., Finney, A., Sauro, H. M., Bolouri, H., Doyle, J. C., Kitano, H., Arkin, A. P., Bornstein, B. J., Bray, D., Cornish-Bowden, A., et al. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btg015 · Cited by 3,060 · see citing works
Self-citationCOPASI—a COmplex PAthway SImulator
Hoops, S., Sahle, S., Gauges, R., Lee, C., Pahle, J., Simus, N., Singhal, M., Xu, L., Mendes, P., & Kummer, U. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl485 · Cited by 2,618 · see citing works
PRISM 4.0: Verification of Probabilistic Real-Time Systems
Kwiatkowska, M., Norman, G., & Parker, D. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22110-1_47 · Cited by 2,400 · see citing works
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
NuSMV 2: An OpenSource Tool for Symbolic Model Checking
Cimatti, A., Clarke, E., Giunchiglia, E., et al. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45657-0_29 · Cited by 1,664 · see citing works
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · see citing works
Gries, D. (1981) · Springer eBooks · doi:10.1007/978-1-4612-5983-1 · Cited by 1,437 · see citing works
COBRApy: COnstraints-Based Reconstruction and Analysis for Python
Ebrahim, A., Lerman, J. A., Palsson, B. O., & Hyduke, D. R. (2013) · BMC Systems Biology · doi:10.1186/1752-0509-7-74 · Cited by 1,378 · see citing works
A Whole-Cell Computational Model Predicts Phenotype from Genotype
Karr, J. R., Sanghvi, J. C., Macklin, D. N., Gutschow, M. V., Jacobs, J. M., Bolival, B., Assad-Garcia, N., Glass, J. I., & Covert, M. W. (2012) · Cell · doi:10.1016/j.cell.2012.05.044 · Cited by 1,300 · cited 1× · see citing works
“However, modelers constructing models that require functionality that is not supported by the existing standards-based tools (Karr et al., 2012; Goldberg et al. 2018) will find the general-purpose approach to be more practical for some stages of the workflow.”
BackgroundSelf-citationThe systems biology graphical notation.
Novère, N. L., Hucka, M., Mi, H., Moodie, S., Schreiber, F., Sorokin, A., Demir, E., Wegner, K., Aladjem, M. I., Wimalaratne, S. M., et al. (2009) · Nature Biotechnology · doi:10.1038/nbt.1558 · Cited by 903 · see citing works
Self-citationTen Simple Rules for Reproducible Computational Research
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003285 · Cited by 880 · see citing works
InChI, the IUPAC International Chemical Identifier
Heller, S. R., Mcnaught, A. D., Плетнев, И. В., Stein, S. E., & Tchekhovskoi, D. V. (2015) · Journal of Cheminformatics · doi:10.1186/s13321-015-0068-4 · Cited by 875 · see citing works
Database resources of the National Center for Biotechnology Information
Coordinators, N. R. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks1189 · Cited by 849 · see citing works
The minimum information about a proteomics experiment (MIAPE)
Taylor, C. F., Paton, N. W., Lilley, K. S., Binz, P., Julian, R. K., Jones, A. R., Zhu, W., Apweiler, R., Aebersold, R. H., Deutsch, E. W., et al. (2007) · Nature Biotechnology · doi:10.1038/nbt1329 · Cited by 769 · see citing works
The BioPAX community standard for pathway data sharing
Demir, E., Cary, M. P., Paley, S., Fukuda, K., Lemer, C., Vastrik, I., Wu, G., D'Eustachio, P., Schaefer, C., Luciano, J., et al. (2010) · Nature Biotechnology · doi:10.1038/nbt.1666 · Cited by 745 · cited 3× · see citing works
“OpenCOR is a modeling environment, which can be used to simulate models described using CellML (Garny and Hunter, 2015).”
BackgroundKidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., et al. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002456 · Cited by 700 · see citing works
Karp, P. D., Paley, S., & Romero, P. (2002) · Bioinformatics · doi:10.1093/bioinformatics/18.suppl_1.s225 · Cited by 661 · see citing works
Li, C., Donizelli, M., Rodriguez, N., Dharuri, H., Endler, L., Chelliah, V., Li, L., He, E., Henry, A., Stefan, M. I., et al. (2010) · BMC Systems Biology · doi:10.1186/1752-0509-4-92 · Cited by 597 · see citing works
Integration from proteins to organs: the Physiome Project
Hunter, P. J., & Borg, T. K. (2003) · Nature Reviews Molecular Cell Biology · doi:10.1038/nrm1054 · Cited by 488 · see citing works
The Ontology for Biomedical Investigations
Bandrowski, A. E., Brinkman, R. R., Brochhausen, M., Brush, M., Bug, B., Chibucos, M. C., Clancy, K., Courtot, M., Derom, D., Dumontier, M., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0154556 · Cited by 387 · see citing works
Database resources of the National Center for Biotechnology Information
Coordinators, N. R. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt1146 · Cited by 385 · see citing works
An Overview of CellML 1.1, a Biological Model Description Language
Cuellar, A. A., Lloyd, C. M., Nielsen, P. F., Bullivant, D. P., Nickerson, D. P., & Hunter, P. J. (2003) · SIMULATION · doi:10.1177/0037549703040939 · Cited by 345 · see citing works
Self-citationWeb-based kinetic modelling using JWS Online
Olivier, B. G., & Snoep, J. L. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth200 · Cited by 326 · see citing works
Systems biology: parameter estimation for biochemical models
Ashyraliyev, M., Fomekong‐Nanfack, Y., Kaandorp, J. A., & Blom, J. G. (2009) · The FEBS Journal · doi:10.1111/j.1742-4658.2008.06844.x · Cited by 323 · see citing works
Controlled vocabularies and semantics in systems biology
Courtot, M., Juty, N., Knüpfer, C., Waltemath, D., Zhukova, A., Dräger, A., Dumontier, M., Finney, A., Golebiewski, M., Hastings, J., et al. (2011) · Molecular Systems Biology · doi:10.1038/msb.2011.77 · Cited by 316 · cited 2× · see citing works
“Use the systems biology ontology (SBO) (Courtot et al., 2011) to precisely record and categorize the semantics of model components, including assumptions, the types of rate laws, and the roles of species in reactions and rate laws.”
MethodThe minimum information required for reporting a molecular interaction experiment (MIMIx)
Orchard, S. E., Salwiński, Ł., Kerrien, S., Montecchi‐Palazzi, L., Oesterheld, M., Stümpflen, V., Ceol, A. G. M., Chatr‐aryamontri, A., Armstrong, J. T., Woollard, P. M., et al. (2007) · Nature Biotechnology · doi:10.1038/nbt1324 · Cited by 280 · see citing works
Waltemath, D., Adams, R., Bergmann, F. T., Hucka, M., Kolpakov, F., Miller, A. K., Moraru, I., Nickerson, D., Sahle, S., Snoep, J., et al. (2011) · BMC Systems Biology · doi:10.1186/1752-0509-5-198 · Cited by 269 · see citing works
Self-citationOpen science is a research accelerator
Woelfle, M., Olliaro, P., & Todd, M. H. (2011) · Nature Chemistry · doi:10.1038/nchem.1149 · Cited by 263 · see citing works
Virtual Cell modelling and simulation software environment
Moraru, I. I., Schaff, J. C., Slepchenko, B. M., Blinov, M. L., Morgan, F., Lakshminarayana, A., Gao, F., Li, Y., & Loew, L. M. (2008) · IET Systems Biology · doi:10.1049/iet-syb:20080102 · Cited by 236 · see citing works
Making scientific computations reproducible
Schwab, M., Karrenbach, N., & Claerbout, J. F. (2000) · Computing in Science & Engineering · doi:10.1109/5992.881708 · Cited by 232 · see citing works
G-quadruplexes are transcription factor binding hubs in human chromatin
Spiegel, J., Cuesta, S. M., Adhikari, S., Hänsel-Hertsch, R., Tannahill, D., & Balasubramanian, S. (2021) · Genome Biology · doi:10.1186/s13059-021-02324-z · Cited by 231 · see citing works
Data2Dynamics: a modeling environment tailored to parameter estimation in dynamical systems
Raue, A., Steiert, B., Schelker, M., Kreutz, C., Maiwald, T., Hass, H., Vanlier, J., Tönsing, C., Adlung, L., Engesser, R., et al. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv405 · Cited by 226 · see citing works
Programming biological models in Python using PySB
Lopez, C. F., Muhlich, J. L., Bachman, J. A., & Sorger, P. K. (2013) · Molecular Systems Biology · doi:10.1038/msb.2013.1 · Cited by 225 · see citing works
BioNetGen 2.2: advances in rule-based modeling
Harris, L. A., Hogg, J., Tapia, J.-J., Sekar, J. A. P., Gupta, S., Korsunsky, I., Arora, A., Barua, D., Sheehan, R. P., & Faeder, J. R. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw469 · Cited by 211 · see citing works
Encyclopedia of Computational Neuroscience
Schwedhelm, P., Treue, S., Jaeger, D., & Jung, R. (2013) · Springer eBooks · doi:10.1007/978-1-4614-7320-6 · Cited by 211 · see citing works
Schmidt, K. (2000) · Lecture notes in computer science · doi:10.1007/3-540-44988-4_27 · Cited by 180 · see citing works
COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project
Bergmann, F. T., Adams, R., Moodie, S., Cooper, J., Glont, M., Golebiewski, M., Hucka, M., Laibe, C., Miller, A. K., Nickerson, D. P., et al. (2014) · BMC Bioinformatics · doi:10.1186/s12859-014-0369-z · Cited by 173 · see citing works
Self-citationMinimum Information About a Simulation Experiment (MIASE)
Waltemath, D., Adams, R., Beard, D. A., Bergmann, F. T., Bhalla, U. S., Britten, R., Chelliah, V., Cooling, M. T., Cooper, J., Crampin, E. J., et al. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001122 · Cited by 168 · see citing works
Self-citationAn invitation to reproducible computational research
Donoho, D. L. (2010) · Biostatistics · doi:10.1093/biostatistics/kxq028 · Cited by 167 · see citing works
Mobley, A. K., Linder, S. K., Braeuer, R. R., Ellis, L. M., & Zwelling, L. A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0063221 · Cited by 164 · see citing works
Tellurium: An extensible python-based modeling environment for systems and synthetic biology
Choi, K., Medley, K., König, M., Stocking, K., Smith, L. P., Gu, S., & Sauro, H. M. (2018) · Biosystems · doi:10.1016/j.biosystems.2018.07.006 · Cited by 164 · see citing works
Self-citationVerification and validation of simulation models
Sargent, R. G. (2009) · IEEE Engineering Management Review · doi:10.1109/emr.2009.5235461 · Cited by 155 · see citing works
libRoadRunner: a high performance SBML simulation and analysis library
Somogyi, E., Bouteiller, J. C., Glazier, J. A., König, M., Medley, K., Swat, M., & Sauro, H. M. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv363 · Cited by 150 · see citing works
Self-citationFAIRDOMHub: a repository and collaboration environment for sharing systems biology research
Wolstencroft, K., Krebs, O., Snoep, J. L., Stanford, N., Bacall, F., Golebiewski, M., Kuzyakiv, R., Nguyen, Q., Owen, S., Soiland‐Reyes, S., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw1032 · Cited by 132 · see citing works
Antimony: a modular model definition language
Smith, L. P., Bergmann, F., Chandran, D., & Sauro, H. M. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp401 · Cited by 128 · see citing works
Self-citationCasadevall, A., & Fang, F. C. (2010) · Infection and Immunity · doi:10.1128/iai.00908-10 · Cited by 124 · see citing works
OpenCOR: a modular and interoperable approach to computational biology
Garny, A., & Hunter, P. J. (2015) · Frontiers in Physiology · doi:10.3389/fphys.2015.00026 · Cited by 116 · see citing works
Deutsch, E. W., Ball, C. A., Berman, J. J., Bova, G. S., Brāzma, A., Bumgarner, R. E., Campbell, D. S., Causton, H. C., Christiansen, J. H., Daian, F., et al. (2008) · Nature Biotechnology · doi:10.1038/nbt1391 · Cited by 116 · see citing works
SEEK: a systems biology data and model management platform
Wolstencroft, K., Owen, S., Krebs, O., Nguyen, Q., Stanford, N., Golebiewski, M., Weidemann, A., Bittkowski, M., An, L., Shockley, D., et al. (2015) · BMC Systems Biology · doi:10.1186/s12918-015-0174-y · Cited by 113 · see citing works
Moreau, L., Groth, P., Cheney, J., Lebo, T., & Miles, S. (2015) · Journal of Web Semantics · doi:10.1016/j.websem.2015.04.001 · Cited by 111 · see citing works
Clarke, E. M., Faeder, J. R., LANGMEAD, C. J., Harris, L. A., Jha, S. K., & Legay, A. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88562-7_18 · Cited by 105 · see citing works
MIRIAM Resources: tools to generate and resolve robust cross-references in Systems Biology
Laibe, C., & Novère, N. G. L. (2007) · BMC Systems Biology · doi:10.1186/1752-0509-1-58 · Cited by 102 · see citing works
How do centralized and distributed version control systems impact software changes?
Brindescu, C., Codoban, M., Shmarkatiuk, S., et al. (2014) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/2568225.2568322 · Cited by 95 · see citing works
RightField: embedding ontology annotation in spreadsheets
Wolstencroft, K., Owen, S., Matthew Horridge, A., Krebs, O., Müller, W., Snoep, J. L., du Preez, F. B., & Goble, C. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr312 · Cited by 83 · see citing works
SBML-PET: a Systems Biology Markup Language-based parameter estimation tool
Zi, Z., & Klipp, E. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl443 · Cited by 83 · see citing works
Emerging whole-cell modeling principles and methods
Goldberg, A. P., Szigeti, B., Yin, H.-C., Sekar, J. A. P., Roth, Y. D., & Karr, J. R. (2017) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2017.12.013 · Cited by 82 · see citing works
Self-citationi B io S im 3: A Tool for Model-Based Genetic Circuit Design
Watanabe, L. H., Nguyen, T., Zhang, M., Zundel, Z., Zhang, Z., Madsen, C. K., Roehner, N., & Myers, C. J. (2018) · ACS Synthetic Biology · doi:10.1021/acssynbio.8b00078 · Cited by 82 · see citing works
Community Curation in Open Dataset Repositories: Insights from Zenodo
Sicilia, M., García‐Barriocanal, E., & Sánchez‐Alonso, S. (2017) · Procedia Computer Science · doi:10.1016/j.procs.2017.03.009 · Cited by 78 · see citing works
Basics of estimating measurement uncertainty.
White, G. (2008) · PubMed · Cited by 76 · see citing works
Ewald, R., & Uhrmacher, A. (2014) · ACM Transactions on Modeling and Computer Simulation · doi:10.1145/2567895 · Cited by 75 · see citing works
Modeling of uncertainties in biochemical reactions
Mišković, L., & Vassily Hatzimanikatis, A. (2010) · Biotechnology and Bioengineering · doi:10.1002/bit.22932 · Cited by 74 · see citing works
A blueprint for human whole-cell modeling
Szigeti, B., Roth, Y. D., Sekar, J. A. P., Goldberg, A. P., Pochiraju, S. C., & Karr, J. R. (2018) · Current Opinion in Systems Biology · doi:10.1016/j.coisb.2017.10.005 · Cited by 72 · see citing works
PyDREAM: high-dimensional parameter inference for biological models in python
Shockley, E. M., Vrugt, J. A., & Lopez, C. F. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx626 · Cited by 71 · see citing works
SBRML: a markup language for associating systems biology data with models
Dada, J. O., Spasic, I., Paton, N., & Mendes, P. (2010) · Bioinform. · doi:10.1093/bioinformatics/btq069 · Cited by 66 · see citing works
Singh, J. (2011) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.81919 · Cited by 64 · see citing works
Stochastic Simulation Service: Bridging the Gap between the Computational Expert and the Biologist
Drawert, B., Hellander, A., Bales, B., Banerjee, D., Bellesia, G., Daigle, B. J., Douglas, G., Gu, M., Gupta, A. K., Hellander, S., et al. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005220 · Cited by 63 · see citing works
Software for Portable Scientific Data Management
Brown, S. A., Folk, M., Goucher, G., et al. (1993) · Computers in Physics · doi:10.1063/1.4823180 · Cited by 57 · see citing works
Pebesma, E. J., Mailund, T., & Hiebert, J. (2016) · The R Journal · doi:10.32614/rj-2016-061 · Cited by 55 · see citing works
Memote: A community driven effort towards a standardized genome-scale metabolic model test suitePreprint
Lieven, C., Beber, M. E., Olivier, B. G., Bergmann, F., Ataman, M., Babaei, P., Bartell, J. A., Blank, L. M., Chauhan, S., Correia, K., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/350991 · Cited by 55 · see citing works
Waltemath, D., & Wolkenhauer, O. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2555481 · Cited by 49 · see citing works
Plan U: Universal access to scientific and medical research via funder preprint mandates
Sever, R., Eisen, M. B., & Inglis, J. R. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000273 · Cited by 49 · see citing works
PyBioNetFit and the Biological Property Specification Language
Mitra, E. D., Suderman, R., Colvin, J. C., Ionkov, A., Hu, A., Sauro, H. M., Posner, R. G., & Hlavacek, W. S. (2019) · iScience · doi:10.1016/j.isci.2019.08.045 · Cited by 45 · see citing works
Self-citationElofsson, A., Hess, B., Lindahl, E. R., Onufriev, A. V., van der Spoel, D., & Wallqvist, A. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006649 · Cited by 43 · see citing works
SBtab: a flexible table format for data exchange in systems biology
Lubitz, T., Hahn, J., Bergmann, F., Noor, Ε., Klipp, E., & Liebermeister, W. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw179 · Cited by 39 · see citing works
Collaborative infrastructure for test-driven scientific model validation
Omar, C., Aldrich, J., & Gerkin, R. C. (2014) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/2591062.2591129 · Cited by 35 · see citing works
Guidelines for reproducibly building and simulating systems biology models
Medley, J., Goldberg, A. P., & Karr, J. R. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2591960 · Cited by 28 · see citing works
Self-citationComputational Methods in Systems Biology
(2007) · Lecture notes in computer science · doi:10.1007/978-3-032-30870-2 · Cited by 24 · see citing works
Schutter, E. D. (2010) · Neuroinformatics · doi:10.1007/s12021-010-9084-8 · Cited by 20 · see citing works
Ten Simple Rules for Experiments’ Provenance
Kazic, T. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004384 · Cited by 20 · see citing works
Metadata and Provenance Management
Deelman, E., Berriman, G. B., Chervenak, A., et al. (2009) · Chapman & Hall/CRC computational science series/Chapman & Hall/CRC computational science · doi:10.1201/9781420069815-c12 · Cited by 15 · see citing works
phraSED-ML: A paraphrased, human-readable adaptation of SED-ML
Choi, K., Smith, L. P., Medley, J., & Sauro, H. M. (2016) · J. Bioinform. Comput. Biol. · doi:10.1142/s0219720016500359 · Cited by 15 · see citing works
Self-citationSEPIO: A semantic model for the integration and analysis of scientific evidence
Brush, M., Shefchek, K., & Haendel, M. A. (2016) · Cited by 13 · see citing works
Haas, P. J., Henderson, S. G., & L’Ecuyer, P. (2014) · ACM Transactions on Modeling and Computer Simulation · doi:10.1145/2555690 · Cited by 10 · see citing works
Executable Simulation Model of the Liver
König, M. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-801238-3.11682-4 · Cited by 10 · see citing works
Huynh, T. D., Groth, P. T., & Zednik, S. (2013) · Cited by 9 · see citing works
Simulation Experiment Description Markup Language (SED-ML)
Waltemath, D., & Novère, N. L. (2014) · Encyclopedia of Computational Neuroscience · doi:10.1007/978-1-4614-7320-6_377-6 · Cited by 8 · see citing works
Executable Simulation Model of the LiverPreprint
König, M. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.01.04.894873 · Cited by 8 · see citing works
Lang, P. F., Chebaro, Y., Zheng, X., Sekar, J. A. P., Shaikh, B., Natale, D., & Karr, J. R. (2019) · Genome Biology · doi:10.1186/s13059-020-02025-z · Cited by 7 · see citing works
Self-citationStructured spreadsheets with ObjTables enable data reuse and integration.
Karr, J. R., Liebermeister, W., Goldberg, A. P., et al. (2020) · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- American Journal of Political Science (AJPS). (2016). American journal of political science qualitative data verification checklist. https://ajps.org/wp-content/uploads/2019/01/ajps-qualdata-checklist-ver-1-0.pdf.
- Ewald R, and Uhrmacher AM (2014). SESSL: a domain-specific language for simulation experiments. ACM Trans. Model. Comput. Simul. 24, 1– 25.
- Rosen L (2005). Open Source Licensing, Volume 692 (Prentice Hall).
- Sherman MA, Middleton JL, Schmidt JP, Paik DS, Blemker SS, Habib AW, Anderson FC, Delp SL, and Altman RB (2005). The simtk framework for physics-based simulation of biological structures: preliminary design. In Proceedings of the workshop on component models and frameworks in high performance computing.
Cited by
Sorted by citations
Reproducibility in systems biology modelling
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2021) · Molecular Systems Biology · doi:10.15252/msb.20209982 · Cited by 131 · see citing works
“In parallel to the reproducibility scorecard, others have recently proposed recommendations for best practices (Porubsky et al , 2020) and specialized peer review of models (Papin et al , 2020).”
Adapting modeling and simulation credibility standards to computational systems biology
Tatka, L. T., Smith, L. P., Hellerstein, J. L., & Sauro, H. M. (2023) · Journal of Translational Medicine · doi:10.1186/s12967-023-04290-5 · Cited by 25 · see citing works
Self-citationRelating simulation studies by provenance—Developing a family of Wnt signaling models
Budde, K., Smith, J., Wilsdorf, P., Haack, F., & Uhrmacher, A. M. (2021) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1009227 · Cited by 23 · see citing works
“(2020) [9] looks at all stages of a biochemical simulation study and at tools supporting their reproducibility.”
BackgroundStandards, dissemination, and best practices in systems biology
Shin, J., Porubsky, V., Carothers, J., & Sauro, H. M. (2023) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2023.102922 · Cited by 15 · see citing works
Self-citation
- Authors
- Porubsky, V. L., Goldberg, A. P., Rampadarath, A. K., Nickerson, D. P., Karr, J. R., & Sauro, H. M.
- Affiliations
- University of Washington (US)
- Icahn School of Medicine at Mount Sinai (US)
- University of Auckland (NZ)
- Corresponding author
- Porubsky, Veronica L.
- Goldberg, Arthur P.
- ORCID
- Venue
- Cell Systems
- Publisher
- Elsevier BV
- Published
- 2020-08
- Volume
- 11
- Issue
- 2
- Pages
- 109-120
- ISSN
- 2405-4712 (ISSN-L)
- PMC
- PMC7480321
- OpenAlex
- W3080314885
- Cited by
- 45
- References
- 96
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 11 · 2023: 10 · 2022: 7 · 2021: 9
- FWCI
- 1.25
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Cell Systems (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- reproducibility, biochemical models, dynamic modeling, model reuse, model sharing, model validation, model simulation, data sharing, standard adoption, model documentation, best practices
- MeSH terms
- Computer Simulation, Humans, Systems Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
