Reproducibility in systems biology modelling
Open accessPublished version of a preprint
The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 3 of the 4 sources its claims cite were checked for support.
Abstract
Reproducibility of scientific results is a key element of science and credibility. The lack of reproducibility across many scientific fields has emerged as an important concern. In this piece, we assess mathematical model reproducibility and propose a scorecard for improving reproducibility in this field.
Overview
CoChat Summary
TL;DR
A systematic BioModels curation study of 455 published ODE models found that 51% directly reproduced at least one main simulation figure, while 49% did not; further effort raised reproducibility to 63%. The authors propose an 8-point reproducibility scorecard for modellers, reviewers, and journal editors.
Key findings
- Of 455 kinetic models, 233 (51%) were directly reproducible; about 49% were not reproducible due to incorrect or missing information in the manuscript. Among 66 models submitted to BioModels in SBML by authors, only 37 could be reproduced directly.
- About 12% of models were reproduced with further effort: 40 models (9%) with manual empirical corrections and 13 models (3%) with author support, bringing total reproducibility to about 63%. The authors contacted corresponding authors for 90 models; fewer than a third (27) responded, and about 70% did not respond.
- Of the 169 non-reproducible models (37%), 99 had identified reasons: missing parameter values (n = 52), missing initial conditions (n = 44), and inconsistency in model structure (n = 36), including combinations; in 70 models the reason was unclear.
- In a randomly selected subset of 110 models, a score ≥ 4 on the 8-point scorecard was significantly associated with reproducibility: χ²(1, N = 110) = 10.0733, P = 0.0015, odds ratio = 4.62 (95% CI: 1.71, 12.44). Questions Q2–Q8 were significantly associated with reproducibility (P < 0.05); Q1 could not be tested because all 110 models scored “yes.”
Methods
- The study systematically analysed 455 published ODE models from 152 journals, representing over 20% of all public literature-based models in BioModels. Models were not randomly sampled but selected based on BioModels curation priorities, including funding, collaborations, curators’ interest, and direct submissions.
- Mathematical equations for 389 models were manually encoded in SBML using COPASI from the original manuscripts; 66 author-submitted SBML models were cross-checked. Simulations were predominantly performed in COPASI, or in other software such as SimBiology, libSBMLsim, and Mathematica when COPASI was used originally.
- A model was considered reproducible if it reproduced at least one main simulation figure using software different from the original, with minor deviations acceptable if they did not affect the scientific conclusion. Models were classified as directly reproducible, reproduced with manual corrections, reproduced with author support, or non-reproducible, and reasons for failure were recorded. The scorecard was tested on 110 randomly selected models using chi-square tests.
Limitations
- The paper states the sample was not randomized and was selected based on BioModels curation priorities, though it covered models from 152 journals. It also states the focus was only on ODE models, one type of kinetic model; other kinetic models are likely affected to the same extent or more, and non-reproducible models were less likely to be selected for curation, so the 49% non-reproducibility rate is likely an underestimate.
- The paper states the reproducibility criterion required reproducing at least one figure, not all relevant figures, as a compromise between reliability and effort. It also notes possible curator error but argues this is unlikely to alter conclusions because curators consult as a team and non-reproducibility was similar among author-submitted SBML models (29 of 66, 44%).
- The paper states author contact was not always feasible due to authors’ change of institutions, change of field, leaving academia, or death, and only 27 of 90 contacted authors responded. It also notes non-reproducible models are not explicitly labelled in BioModels because failure could be due to curator error and to avoid discouraging authors from sharing models.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
3 Valid · 0 Not valid · 1 Not checked · 44 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, mathematical models from a number of published articles were shown not to reproduce the simulation results described in the article (Mendes, 2018).”
§ Reproducibility in systems biology modelling > Abstract
What the sources say
- Partly supportedcites Mendes, 2018 sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says many computational articles are hard or impossible to reproduce, and Some recommendations (page unavailable) reports that only a minority of roughly 400 surveyed biomodel articles described their work sufficiently for reproduction. These passages support a reproducibility problem, but do not say that the mathematical models themselves were tested and shown to produce results different from those reported. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similar to the curation service in BioModels, JWSOnline and the Physiome repository through the Centre for Reproducible BioMedical modelling (Papin et al, 2020) provide expert curation services to validate published reports.”
§ Reproducibility in systems biology modelling > Reproducibility scorecard > Modelling guidelines/community efforts
What the sources say
- Partly supportedcites Papin et al, 2020 sourceExperimental — accuracy not yet measured
In the main text (page not specified), Papin et al. describe a partnership with the Center for Reproducible Biomedical Modeling for peer review of computational-model reproducibility. Our copy does not mention JWSOnline or the Physiome repository, or establish that either provides expert curation services to validate published reports. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“BioModels, being the largest repository of open, curated, well‐annotated and findable models, contributes significantly to reproducibility (Mendes, 2018).”
§ Reproducibility in systems biology modelling > Reproducibility scorecard > Modelling guidelines/community efforts
What the sources say
- Partly supportedcites Mendes, 2018 sourceExperimental — accuracy not yet measured
In “Publication of electronic materials” (page unavailable), Mendes calls BioModels the most widely used model repository and says its curators verify published results, making submission a major check on reproducibility. The section also says public repositories help materials remain findable. It does not establish that BioModels is the largest repository or that its models are well annotated. (LLM, conf=0.85)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Forty models (9%) were successfully reproduced with manual empirical correction of the inaccurate reporting in the manuscript.”
§ Reproducibility in systems biology modelling > 12% of the models could be reproduced with further effort
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“About half of the models of authors who responded (13 models, or 3% of the total number of models) were subsequently reproduced.”
§ Reproducibility in systems biology modelling > 12% of the models could be reproduced with further effort
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 48 claims and every source verdict.
- It may state two separate claims as one. 12
- The quote only partly states this claim. 1
- Probably not a substantial claim. 9
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It was recently discovered that 49% of published models undergoing the review and curation process for the BioModels [6] database were not reproducible primarily due to missing materials necessary for simulation, the availability of the model and code in public databases, and lack of documentation [7].”
From Adapting modeling and simulation credibility standards to computational systems biology (2023) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“With some extra effort, an additional 12% of the published models could be reproduced.”
From Adapting modeling and simulation credibility standards to computational systems biology (2023) · cites this paper as [7]
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentχ²(1) = 10.0733reported p = .0015 · recomputed p = .002
“X2 (1, N = 110) = 10.0733, p = .0015”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Self-check results appear here once an independent check has confirmed them.
Citations
Cites
Gene Ontology: tool for the unification of biology
Ashburner, M., Ball, C. A., Blake, J. A., Botstein, D., Butler, H., Cherry, J. M., Davis, A. P., Dolinski, K., Dwight, S. S., Eppig, J. T., et al. (2000) · Nature Genetics · doi:10.1038/75556 · Cited by 43,206
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
UniProt: a worldwide hub of protein knowledge
The UniProt Consortium (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1049 · Cited by 7,698
The Gene Ontology Resource: 20 years and still GOing strong
The Gene Ontology Consortium (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1055 · Cited by 4,747
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
COPASI—a COmplex PAthway SImulator
Hoops, S., Sahle, S., Gauges, R., Lee, C., Pahle, J., Simus, N., Singhal, M., Xu, L., Mendes, P., & Kummer, U. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl485 · Cited by 2,618
The reactome pathway knowledgebase
Jassal, B., Matthews, L., Viteri, G., Gong, C., Lorente, P., Fabregat, A., Sidiropoulos, K., Cook, J., Gillespie, M., Haw, R., et al. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz1031 · Cited by 2,159
Self-citationFederhen, S. (2012) · Nucleic Acids Research · doi:10.1093/nar/gkr1178 · Cited by 1,436
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · cited 2×
“Although the lack of reproducibility has been discussed within the community (Mendes, 2018; Papin et al , 2020), it was not expected to be this adverse.”
Cited by
Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers
Wratten, L., Wilm, A., & Göke, J. (2021) · Nature Methods · doi:10.1038/s41592-021-01254-9 · Cited by 225
Musuamba, F. T., Skottheim Rusten, I., Lesage, R., Russo, G., Bursi, R., Emili, L., Wangorsch, G., Manolis, E., Karlsson, K. E., Kulesza, A., et al. (2021) · CPT: Pharmacometrics & Systems Pharmacology · doi:10.1002/psp4.12669 · Cited by 164
To Sobol or not to Sobol? The effects of sampling schemes in systems biology applications
Renardy, M., Joslyn, L. R., Millar, J. A., & Kirschner, D. E. (2021) · Mathematical Biosciences · doi:10.1016/j.mbs.2021.108593 · Cited by 182
The European Bioinformatics Institute (EMBL-EBI) in 2021
Cantelli, G., Bateman, A., Brooksbank, C., Petrov, A. I., Malik-Sheriff, R. S., Ide-Smith, M., Hermjakob, H., Flicek, P., Apweiler, R., Birney, E., et al. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkab1127 · Cited by 96
Self-citationImmune digital twins for complex human pathologies: applications, limitations, and challenges
Niarakis, A., Laubenbacher, R., An, G. C., Ilan, Y., Fisher, J., Flobak, Å., Reiche, K., Martínez, M. R., Geris, L., Ladeira, L., et al. (2024) · npj Systems Biology and Applications · doi:10.1038/s41540-024-00450-5 · Cited by 127
Self-citationBoolean modelling as a logic-based dynamic approach in systems medicine
Hemedan, A. A., Niarakis, A., Schneider, R., & Ostaszewski, M. (2022) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2022.06.035 · Cited by 56
“Another example is the reproducibility scorecard, a list of eight questions to evaluate the reusability and reproducibility of the systems biology models [124].”
BackgroundFrechen, S., & Rostami‐Hodjegan, A. (2022) · Pharmaceutical Research · doi:10.1007/s11095-022-03250-w · Cited by 70
“The issue of reproducibility has been subject of several recent data analyses of models in the space of systems biology (24) with indications that these may not be satisfactory without all the quality assurance steps outlined above.”
BackgroundDuong‐Trung, N., Born, S., Kim, J. W., Schermeyer, M., Paulick, K., Borisyak, M., Bournazou, M. N. C., Werner, T., Scholz, R., Schmidt-Thieme, L., et al. (2022) · Biochemical Engineering Journal · doi:10.1016/j.bej.2022.108764 · Cited by 67
Puniya, B. L., Verma, M., Damiani, C., Bakr, S. H., & Dräger, A. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae090 · Cited by 57
“…(help with manual curation of the models), and the (iii) ‘COmputational Modeling in BIology Network’ (COMBINE) (http://co.mbine.org/) (help annotate and semantically enrich the models) have been instrumental (Malik-Sheriff et al. , 2020, Tiwari et al. , 2021, K € onig et al. , 2023).”
MethodBioUML—towards a universal research platform
Kolpakov, F., Akberdin, I. R., Kiselev, I. N., Kolmykov, S. K., Kondrakhin, Y. V., Kulyashov, M. A., Kutumova, E. O., Pintus, S. S., Ryabova, A., Sharipov, R., et al. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac286 · Cited by 53
Combining hypothesis- and data-driven neuroscience modeling in FAIR workflows
Eriksson, O., Bhalla, U. S., Blackwell, K. T., Crook, S. M., Keller, D. X., Kramer, A., Linne, M., Saudargienė, A., Wade, R. C., & Kotaleski, J. H. (2022) · eLife · doi:10.7554/elife.69013 · Cited by 41
Kutumova, E. O., Akberdin, I. R., Kiselev, I. N., Sharipov, R., Egorova, V. S., Syrocheva, A. O., Parodi, A., Zamyatnin, A. A., & Kolpakov, F. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms232012560 · Cited by 44
Making in silico predictive models for toxicology FAIR
Cronin, M., Belfield, S. J., Briggs, K. A., Enoch, S. J., Firman, J. W., Frericks, M., Garrard, C., Maccallum, P. H., Madden, J. C., Pastor, M., et al. (2023) · Regulatory Toxicology and Pharmacology · doi:10.1016/j.yrtph.2023.105385 · Cited by 43
Implementing code review in the scientific workflow: Insights from ecology and evolutionary biology
Ivimey‐Cook, E. R., Pick, J. L., Bairos‐Novak, K. R., Čulina, A., Gould, E., Grainger, M., Marshall, B. M., Moreau, D., Paquet, M., Royauté, R., et al. (2023) · Journal of Evolutionary Biology · doi:10.1111/jeb.14230 · Cited by 48
“As several recent papers have highlighted (Archmiller et al., 2020 ; Errington et al., 2021 ; Mi-nocher et al., 2021 ; Obels et al., 2020 ; Tiwari et al., 2021 ), reproducibility in research results is often very low.”
BackgroundUnifying the identification of biomedical entities with the Bioregistry
Hoyt, C. T., Balk, M. A., Callahan, T. J., Domingo‐Fernándéz, D., Haendel, M. A., Hegde, H., Himmelstein, D. S., Karis, K., Kunze, J., Lubiana, T., et al. (2022) · Scientific Data · doi:10.1038/s41597-022-01807-3 · Cited by 44
“This vision aligns well with the recommendations from a recent assessment of the reproduciblity of such models 55 that highlighted the more general importance of annotation using high-quality controlled vocabularies like GO and ChEBI.”
MethodNiarakis, A., Waltemath, D., Glazier, J. A., Schreiber, F., Keating, S., Nickerson, D. P., Chaouiya, C., Siegel, A., Noël, V., Hermjakob, H., et al. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac212 · Cited by 36
“The team developing the BioModels database of mathematical models of biological systems [2] has attempted to systematically reproduce 455 kinetic models published in peer-reviewed research articles from 152 journals [70].”
BackgroundMethodSelf-citationIntegrative systems biology approaches for analyzing microbiome dysbiosis and species interactions
Rehman, S. S. U., Nasar, M. I., Mesquita, C. S., Khodor, S. A., Notebaart, R. A., Ott, S., Mundra, S., Arasardanam, R. P., Muhammad, K., & Alam, M. T. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf323 · Cited by 36
“Furthermore, the experimental validation of model predictions is difficult due to the lack of direct measurements in microbial ecosystems [298, 300].”
Mapping miRNA Research in Schizophrenia: A Scientometric Review
Lim, M., Carollo, A., Neoh, M. J. Y., & Esposito, G. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms24010436 · Cited by 27
Computational models in systems biology: standards, dissemination, and best practices
Vieira, L. S., & Laubenbacher, R. (2022) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2022.102702 · Cited by 25
Adapting modeling and simulation credibility standards to computational systems biology
Tatka, L. T., Smith, L. P., Hellerstein, J. L., & Sauro, H. M. (2023) · Journal of Translational Medicine · doi:10.1186/s12967-023-04290-5 · Cited by 27
“model and code in public databases, and lack of documentation [7].”
BackgroundIn Silico Clinical Trials: Is It Possible?
Arsène, S., Parès, Y., Tixier, E., Granjeon-Noriot, S., Martin, B., Bruezière, L., Couty, C., Courcelles, E., Kahoul, R., Pitrat, J., et al. (2023) · Methods in molecular biology · doi:10.1007/978-1-0716-3449-3_4 · Cited by 26
Automatic Reuse, Adaption, and Execution of Simulation Experiments via Provenance Patterns
Wilsdorf, P., Wolpers, A., Hilton, J., Haack, F., & Uhrmacher, A. M. (2022) · ACM Transactions on Modeling and Computer Simulation · doi:10.1145/3564928 · Cited by 26
Microbiome modeling: a beginner's guide
Lange, E., Kranert, L., Krüger, J., Benndorf, D., & Heyer, R. (2024) · Frontiers in Microbiology · doi:10.3389/fmicb.2024.1368377 · Cited by 21
Standard-GEM supports reproducible development of genome-scale metabolic modelsPreprint
Anton, M., Almaas, E., Bäuerle, F., Beber, M. E., Benfeitas, R., Benito-Vaquerizo, S., Blank, L. M., Clasen, F., de Buck, C., Dräger, A., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.03.21.512712 · Cited by 24
Pre-registration of mathematical models
Ioannidis, J. P. A. (2022) · Mathematical Biosciences · doi:10.1016/j.mbs.2022.108782 · Cited by 20
Toward a common standard for data and specimen provenance in life sciences
Wittner, R., Holub, P., Mascia, C., Frexia, F., Müller, H., Plass, M., Allocca, C. M., Betsou, F., Burdett, T., Cancio, I., et al. (2023) · Learning Health Systems · doi:10.1002/lrh2.10365 · Cited by 30
“1-11 At the same time, the origin and history of specimens used to generate research data often remain inexplicit.”
BackgroundModel Integration in Computational Biology: The Role of Reproducibility, Credibility and Utility
Karr, J. R., Malik‐Sheriff, R. S., Osborne, J. M., González‐Parra, G., Forgoston, E., Bowness, R., Liu, Y., Thompson, R. N., Garira, W., Barhak, J., et al. (2022) · Frontiers in Systems Biology · doi:10.3389/fsysb.2022.822606 · Cited by 21
“In a recent study (Tiwari et al., 2021), the BioModels group analyzed 455 kinetic models published in 152 peer-reviewed journals, a collective work of about 1,400 scientists from 49 countries.”
BackgroundSelf-citationRostami‐Hodjegan, A., & Bois, F. Y. (2021) · CPT Pharmacometrics & Systems Pharmacology · doi:10.1002/psp4.12615 · Cited by 19
“A recent systematic attempt by Tiwari et al. 20 regarding reproducing 455 kinetic models of biological processes published in peer-reviewed research articles revealed that about half (49”
BackgroundQuantitative systems toxicology: modelling to mechanistically understand and predict drug safety
Goldring, C. E., Russomanno, G., Pin, C., Trairatphisan, P., Beattie, K. A., Fisher, C., Piñero, J., Brennan, R., Clausznitzer, D., Copple, I. M., et al. (2025) · Nature Reviews Drug Discovery · doi:10.1038/s41573-025-01308-z · Cited by 27
“Repurposing existing models is an efficient use of current knowledge, but it relies on the robust reporting and reproducibility of published systems models 23 .”
Self-citationThe Role of Model Master Files for Sharing, Acceptance, and Communication with FDA
Fang, L., Gong, Y., Hooker, A. C., et al. (2024) · The AAPS Journal · doi:10.1208/s12248-024-00897-8 · Cited by 18
“Major reproducibility issues have recently been associated with OSC models in the space of systems biology (14).”
Data integration in logic-based models of biological mechanisms
Hall, B. A., & Niarakis, A. (2021) · Current Opinion in Systems Biology · doi:10.1016/j.coisb.2021.100386 · Cited by 24
Akberdin, I. R., Kiselev, I. N., Pintus, S. S., Sharipov, R., Vertyshev, A. Y., Виноградова, О. Л., Popov, D. V., & Kolpakov, F. (2021) · International Journal of Molecular Sciences · doi:10.3390/ijms221910353 · Cited by 22
Schölzel, C., Blesius, V., Ernst, G., & Dominik, A. (2021) · npj Systems Biology and Applications · doi:10.1038/s41540-021-00182-w · Cited by 17
“A recent study by curators of the BioModels database showed that only 51% of 455 published models were directly reproducible 8 .”
BackgroundInfluentialWriting Accessible Theory in Ecology and Evolution: Insights from Cognitive Load Theory
Ou, W. J.-A., Henriques, G. J. B., Senthilnathan, A., Ke, P., Grainger, T. N., & Germain, R. M. (2021) · BioScience · doi:10.1093/biosci/biab133 · Cited by 16
Guebila, M. B., Weighill, D. A., Lopes‐Ramos, C. M., et al. (2022) · Nature Methods · doi:10.1038/s41592-022-01479-2 · Cited by 16
Dray, K. E., Muldoon, J. J., Mangan, N. M., Bagheri, N., & Leonard, J. N. (2022) · ACS Synthetic Biology · doi:10.1021/acssynbio.1c00528 · Cited by 14
Aldibani, H. K. A., Rajput, A. J., & Rostami‐Hodjegan, A. (2023) · Biopharmaceutics & Drug Disposition · doi:10.1002/bdd.2360 · Cited by 14
“Attempts to directly reproduce models, developed on OSC platforms, do not create a good perspective on the reproducibility of OSC modality in QSP and systems biology models (Tiwari et al., 2021).”
MethodAdvancing reproducibility can ease the ‘hard truths’ of synthetic biology
Lux, M. W., Strychalski, E. A., & Vora, G. J. (2023) · Synthetic Biology · doi:10.1093/synbio/ysad014 · Cited by 19
Practical resources for enhancing the reproducibility of mechanistic modeling in systems biology
Blinov, M. L., Gennari, J. H., Karr, J. R., Moraru, I. I., Nickerson, D. P., & Sauro, H. M. (2021) · Current Opinion in Systems Biology · doi:10.1016/j.coisb.2021.06.001 · Cited by 17
Making PBPK models more reproducible in practice
Domínguez-Romero, E., Mazurenko, S., Scheringer, M., dos Santos, V. A. P. M., Evelo, C. T., Anton, M., Hancock, J. M., Županič, A., & Diez, M. S. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae569 · Cited by 16
“[9, 10, 18, 20] Provenance and understandability • Annotations specified in Table 5, • Other annotations, such as text definitions of model components, sources and others, in order to facilitate the code understandability for the users.”
BackgroundMethodInfluentialMetaboverse enables automated discovery and visualization of diverse metabolic regulatory patterns
Berg, J. A., Zhou, Y., Ouyang, Y., Cluntun, A. A., Waller, T. C., Conway, M. E., Nowinski, S. M., Ry, T. V., George, I. D., Cox, J. E., et al. (2023) · Nature Cell Biology · doi:10.1038/s41556-023-01117-9 · Cited by 22
“Metaboverse curates a reaction network database on the basis of a Reactome knowledgebase 17–19 , BiGG 25 or BioModels 26,27 network.”
MethodChung, D., Bakshi, S., & van der Graaf, P. H. (2023) · Pharmaceutics · doi:10.3390/pharmaceutics15030918 · Cited by 12
Squaring a Circle: To What Extent Are Traditional Circuit Analogies Impeding Synthetic Biology?
Ilia, K., & Vecchio, D. D. (2022) · GEN Biotechnology · doi:10.1089/genbio.2021.0014 · Cited by 14
Statistical Perspectives on Reproducibility: Definitions and Challenges
Simkus, A., Coolen‐Maturi, T., Coolen, F. P. A., & Bendtsen, C. (2025) · Journal of Statistical Theory and Practice · doi:10.1007/s42519-025-00459-x · Cited by 15
“Tiwari et al. [146] limited the scope of their paper to systems biology modelling but this idea could also be used in other scientific areas.”
InfluentialRajput, A. J., Aldibani, H. K. A., & Rostami‐Hodjegan, A. (2023) · Biopharmaceutics & Drug Disposition · doi:10.1002/bdd.2357 · Cited by 13
FROG Analysis Ensures the Reproducibility of Genome Scale Metabolic ModelsPreprint
Raman, K., Kratochvíl, M., Olivier, B. G., König, M., Sengupta, P., Baskaran, D. K. K., Nguyen, T. V. N., Lobo, D., Wilken, S. E., Tiwari, K. K., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.24.614797 · Cited by 10
“A study (Tiwari et al., 2021) highlighted that approximately half of the selected ordinary differential equation (ODE) models published in peer-reviewed journals could not be reproduced using the information provided in the publications.”
BackgroundSelf-citationvan Kampen, A. H. C., Mahamune, U., Jongejan, A., van Schaik, B. D. C., Balashova, D., Lashgari, D., Pras‐Raves, M. L., Wever, E. J. M., Dane, A. D., García-Valiente, R., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-52446-8 · Cited by 24
Waites, W., Cavaliere, M., Manheim, D., Panovska‐Griffiths, J., & Danos, V. (2021) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2021.110851 · Cited by 15
Openness and Computational Reproducibility in Plant Pathology: Where We Stand and a Way Forward
Sparks, A., Ponte, E. M. D., Alves, K. S., Foster, Z., & Grünwald, N. J. (2023) · Phytopathology · doi:10.1094/phyto-10-21-0430-per · Cited by 11
Systems Biology in ELIXIR: modelling in the spotlight
dos Santos, V. A. P. M., Anton, M., Szomolay, B., Ostaszewski, M., Arts, I. C. W., Benfeitas, R., Angel, V. D. D., Domínguez-Romero, E., Ferk, P., Fey, D., et al. (2022) · F1000Research · doi:10.12688/f1000research.126734.1 · Cited by 14
Self-citationA Guide in Synthetic Biology: Designing Genetic Circuits and Their Applications in Stem Cells
Elnaggar, K. S., Gamal, O., Hesham, N., Ayman, S., Mohamed, N., Moataz, A., Elzayat, E. M., & Hassan, N. (2025) · SynBio · doi:10.3390/synbio3030011 · Cited by 11
White, J. W., Boote, K. J., Kimball, B. A., Porter, C. H., Salmerón, M., Shelia, V., Thorp, K. R., & Hoogenboom, G. (2025) · npj Sustainable Agriculture · doi:10.1038/s44264-025-00067-z · Cited by 13
“Comparing 455 models of biological processes expressed using Systems Biology Markup Language (SBML), Tiwari and co-workers were unable to reproduce results from half of the models 44 .”
Chen, Z., & Wang, L. (2024) · Biotechnology Advances · doi:10.1016/j.biotechadv.2024.108470 · Cited by 11
Deciphering signal transduction networks in the liver by mechanistic mathematical modelling
D’Alessandro, L. A., Klingmüller, U., & Schilling, M. (2022) · Biochemical Journal · doi:10.1042/bcj20210548 · Cited by 6
Large-scale knowledge graph representations of disease processes
Hoch, M., Gupta, S. K., & Wolkenhauer, O. (2024) · Current Opinion in Systems Biology · doi:10.1016/j.coisb.2024.100517 · Cited by 9
Nègre, D., Larhlimi, A., & Bertrand, S. (2023) · PLoS ONE · doi:10.1371/journal.pone.0289757 · Cited by 9
“49 The recurrent non-reproducibility of results, known as the reproducibility crisis [75], 943 affects all areas of science and in silico modelling is no exception [76].”
Sychterz, C. J., Gardner, I. B., Chiang, M., Ramakrishna, R., Neuhoff, S., Perera, V., Merali, S. J., Schmidt, B. J., & Gaohua, L. (2022) · Metabolites · doi:10.3390/metabo12101001 · Cited by 8
Network Reconstruction and Modelling Made Reproducible with moped
Saadat, N. P., van Aalst, M., & Ebenhöh, O. (2022) · Metabolites · doi:10.3390/metabo12040275 · Cited by 8
Bayesian estimation reveals that reproducible models in Systems Biology get more citations
Höpfl, S., Pleiss, J., & Radde, N. E. (2023) · Scientific Reports · doi:10.1038/s41598-023-29340-2 · Cited by 10
“Data D consist of citation counts of models classified by Tiwari et al. 25 into reproducible (Dataset 1) and non-reproducible models (Dataset 2).”
BackgroundMethodInfluentialDémaris, A., Widigson, E. S. K., Ilvemark, J. F. K. F., Steenholdt, C., Seidelin, J. B., Huisinga, W., Michelet, R., Aulin, L. B. S., & Kloft, C. (2022) · Pharmaceutics · doi:10.3390/pharmaceutics14102095 · Cited by 7
“In a recent review, Tiwari et al. (2021) reported that only 49% of 455 published models were directly reproducible using the code and information available in the publication [34].”
BackgroundThe comfort of automation: why cognitive sovereignty matters in AI-driven life sciences
Branda, F., & Ciccozzi, M. (2026) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2026.100158 · Cited by 14
Metabolic modelling: Insights into the machine room of plant metabolism
Machado, T. M., Töpfer, N., & Soltani, F. (2025) · Journal of Plant Physiology · doi:10.1016/j.jplph.2025.154584 · Cited by 9
LimbNET: collaborative platform for simulating spatial patterns of gene networks in limb developmentPreprint
Matyjaszkiewicz, A., & Sharpe, J. A. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.08.07.607075 · Cited by 6
Höpfl, S., Albadry, M., Dahmen, U., Herrmann, K., Kindler, E. M., König, M., Reichenbach, J. R., Tautenhahn, H., Wei, W., Zhao, W.-T., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae312 · Cited by 5
“Importantly, BayModTS contributes to reproducibility according to the FAIR principles by using established tools of the systems biology community and adhering to standard reporting guidelines (Tiwari et al. 2021).”
MethodReproducibility and FAIR principles: the case of a segment polarity network model
Mendes, P. (2023) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2023.1201673 · Cited by 5
“It has been noted that reproducing results from computational studies in general (Mesirov, 2010; Peng, 2011; Stodden et al., 2016), and also computatational systems biology (Waltemath and Wolkenhauer, 2016; Mendes, 2018; Tiwari et al., 2021), is as hard as with laboratory experiments.”
Tiwari, D. D., Hoffmann, N., Didi, K., Deshpande, S., Ghosh, S., Nguyen, T. V. N., Raman, K., Hermjakob, H., & Malik‐Sheriff, R. S. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.05.22.540599 · Cited by 6
Self-citationImplementing Code Review in the Scientific Workflow: Insights from Ecology and Evolutionary BiologyPreprint
Ivimey‐Cook, E. R., Pick, J. L., Bairos‐Novak, K. R., Čulina, A., Gould, E., Grainger, M., Marshall, B. M., Moreau, D., Paquet, M., Royauté, R., et al. (2023) · doi:10.32942/x2cg64 · Cited by 5
Berg, J. A., Zhou, Y., Ouyang, Y., Waller, T. C., Cluntun, A. A., Conway, M. E., Nowinski, S. M., Ry, T. V., George, I., Cox, J. E., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.06.25.171850 · Cited by 6
Gopukumar, S. T., Dwivedi, D., Rahamathulla, M., Ahmed, M. M., Soni, T. K., Natarajan, P. G., Ramkanth, S., Mitra, A., & Das, U. (2026) · Functional & Integrative Genomics · doi:10.1007/s10142-026-01890-4 · Cited by 10
“Initiatives are already underway to establish community-driven libraries of grammar-encoded models, analogous to open-source software repositories (Tiwari et al. 2021).”
Reproducibility and Environmental Efficiency of Metabolomics Cancer Modeling
Jean-Quartier, C., Tscheppe, N., Millonig, S., Klambauer, L., Holzinger, A., Stryeck, S., & Jeanquartier, F. (2026) · Applied Sciences · doi:10.3390/app16020588 · Cited by 7
Building immune digital twins: An international and transdisciplinary community effort
Niarakis, A., An, G. C., Ladeira, L., Hiroi, N. F., Papadopoulou, A., Crawley, F. P., Nikaein, N., Calzone, L., Tsirvouli, E., Balcı, H., et al. (2025) · ImmunoInformatics · doi:10.1016/j.immuno.2025.100060 · Cited by 6
Self-citationMiroshnichenko, M. I., Kolpakov, F., & Akberdin, I. R. (2025) · Viruses · doi:10.3390/v17050589 · Cited by 7
“To address the common issue of model reproducibility in systems biology [40], we additionally reproduced the model using Julia programming language and conducted all equivalent investigations.”
Chen, A., Ren, Q., Zhou, T., Burrage, P., Tian, T., & Burrage, K. (2024) · The Journal of Chemical Physics · doi:10.1063/5.0187202 · Cited by 5
FAIR Sharing of Reproducible Models of Epidemic and Pandemic ForecastPreprint
Ramachandran, K., König, M., Scharm, M., Nguyen, T. V. N., Hermjakob, H., Waltemath, D., & Malik‐Sheriff, R. S. (2022) · Preprints.org · doi:10.20944/preprints202206.0137.v1 · Cited by 4
“Furthermore, misprints, typographical errors, missing or incomplete information, mismatch between the code and the description in the publication create more obstacles to reproduce, reuse, and repurpose models (8).”
Self-citationA toolkit for enhanced reproducibility of RNASeq analysis for synthetic biologists
Garcia, B. J., Urrutia, J. A., Zheng, G., et al. (2022) · Synthetic Biology · doi:10.1093/synbio/ysac012 · Cited by 5
Schölzel, C., Blesius, V., Ernst, G., Goesmann, A., & Dominik, A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0254749 · Cited by 9
“For example, the aforementioned reproducibility study of models in the BioModels database found very similar errors and reproducibility hurdles in half of the 455 examined models [27].”
BackgroundResultSardar, S., Dash, S., Roychowdhury, P., Solanki, J., Sikdar, P., Thangappan, J., & Dutta, S. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag269 · Cited by 13
“Similarly, DRAMMA predicts resistance genes lacking detectable sequence similarity to known ARGs by incorporating protein properties, genomic context, and evolutionary signals [ 49, 50].”
From FAIR to CURE: guidelines for computational models of biological systems
Sauro, H. M., Agmon, E., Blinov, M. L., Gennari, J. H., Hellerstein, J. L., Heydarabadipour, A., Jardine, B., May, E. E., Nickerson, D. P., Smith, L. P., et al. (2026) · npj Systems Biology and Applications · doi:10.1038/s41540-026-00651-0 · Cited by 32
“However, it has also been dis-coveredthatcomputationalexperimentsareoftennotreproducible 15 .”
Self-citationReproducible Boolean model analyses and simulations with the CoLoMoTo software suite: a tutorial
Noël, V., Naldi, A., Calzone, L., et al. (2025) · Interface Focus · doi:10.1098/rsfs.2025.0002 · Cited by 4
“However, like experimental biologists, modellers are facing serious reproducibility challenges, which have been stressed in recent years (see [10], focusing on modelling studies using ordinary differential equations).”
Pelot, N. A., Wang, B., Marshall, D. P., Hussain, M. A., Musselman, E. D., Yu, G. J., Dale, J., Baumgart, I. W., Dardani, D., Zamani, P. T., et al. (2025) · Journal of Neural Engineering · doi:10.1088/1741-2552/adb997 · Cited by 6
“…computing literature, and we added a sixth ‘R’ for ‘readability’; we primarily lever-aged the terms reported in Benureau and Rougier [4] but used the definitions for replicability and reproducibility provided by Miłkowski et al [17] as they are more consistent with the wider literature [18–20].”
Liu, J., Yang, L., Xiao, Q., Li, Z., Cheng, J.-Y., Chen, Z., Zhou, J., Wan, X., Chang, T., Zhang, X., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.17.612997 · Cited by 3
González‐Parra, G., Villanueva-Oller, J., Navarro-González, F. J., Ceberio, J., & Luebben, G. (2024) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2024.114695 · Cited by 5
A Practical Guide to Reproducible Modeling for Biochemical Networks
Porubsky, V. L., & Sauro, H. M. (2023) · Methods in molecular biology · doi:10.1007/978-1-0716-3008-2_5 · Cited by 3
Parker, C., Nelson, E., & Zhang, T. (2022) · Entropy · doi:10.3390/e24121747 · Cited by 4
Reproducible Boolean model analyses and simulations with the CoLoMoTo software suite: a tutorialPreprint
Noël, V., Naldi, A., Calzone, L., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.01.26.634971 · Cited by 3
Tan, N., Assogba, K., Ashworth, W. J., Bogale, B., Cappello, F., Rafique, M. M., Taufer, M., & Nicolae, B. (2024) · International Middleware Conference · doi:10.1145/3652892.3700780 · Cited by 6
Reidelbach, M., Ferrer, E., & Weber, M. (2023) · Proceedings of the Conference on Research Data Infrastructure · doi:10.52825/cordi.v1i.254 · Cited by 3
“Increasing reports about reproduction problems [1], [2] impair the scientific credibility and limit synergies arising from reusing and advancing scientific work.”
BackgroundThe in silico lab: Improving academic code using lessons from biology
Cain, J. Y., Yu, J. S., & Bagheri, N. (2023) · Cell Systems · doi:10.1016/j.cels.2022.11.006 · Cited by 3
Unifying the Identification of Biomedical Entities with the BioregistryPreprint
Hoyt, C. T., Balk, M. A., Callahan, T. J., Domingo‐Fernándéz, D., Haendel, M. A., Hegde, H., Himmelstein, D., Karis, K., Kunze, J., Lubiana, T., et al. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.07.08.499378 · Cited by 3
Dray, K. E., Muldoon, J. J., Mangan, N. M., et al. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.10.20.465216 · Cited by 3
“The workflow in this tutorial introduces and uses aspects of model development that should improve the rigor and reproducibility of modeling in synthetic biology, which represents an open opportunity and need 16, 17 .”
BackgroundMethodThe Practice of Ensuring Repeatable and Reproducible Computational ModelsPreprint
Sauro, H. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05386 · Cited by 4
“A recent survey by the Biomodels team [75] confirmed previous anecdotal evidence that a large proportion (over 50”
BackgroundFAIRification of computational models in biologyPreprint
Balaur, I., Nickerson, D. P., Welter, D., Wodke, J. A. H., Ancien, F., Gebhardt, T., Grouès, V., Hermjakob, H., König, M., Radde, N. E., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.03.21.644517 · Cited by 2
“The BioModels team, for example, has developed a distinct checklist to address the high rate of irreproducible simulation studies 3 .”
Self-citationData Hazards as an ethical toolkit for neurosciencePreprint
García, S. R., Welsh, C., Cara, N. H. D., et al. (2024) · doi:10.31219/osf.io/yn2j9 · Cited by 2
Rahul, R., Stinchcombe, A. R., Joseph, J. W., & Ingalls, B. (2023) · IET Systems Biology · doi:10.1049/syb2.12077 · Cited by 2
“[32–34].”
van Kampen, A. H. C., Mahamune, U., Jongejan, A., van Schaik, B. D. C., Balashova, D., Lashgari, D., Pras‐Raves, M. L., Wever, E. J. M., Dane, A. D., García-Valiente, R., et al. (2023) · doi:10.31222/osf.io/m9yqz · Cited by 2
Questions, data and models underpinning metabolic engineering
van Rosmalen, R. P., dos Santos, V. A. P. M., & Diez, M. S. (2022) · Frontiers in Systems Biology · doi:10.3389/fsysb.2022.998048 · Cited by 3
Schölzel, C., Blesius, V., Ernst, G., et al. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.02.19.431951 · Cited by 2
Rostami‐Hodjegan, A. (2025) · Drug Metabolism and Pharmacokinetics · doi:10.1016/j.dmpk.2025.101514 · Cited by 2
The Virtual Brain Ontology: A Digital Knowledge Framework for Reproducible Brain Network ModelingPreprint
Martin, L., Bülau, K., Pille, M., Schmitt, R. A., Huettl, C., Meier, J. M., Taher, H., Perdikis, D., Schirner, M., Stefanovski, L., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.11.19.689211 · Cited by 1
“When models are published, reproducibility remains limited, partly because of incomplete model descriptions, missing parameter values, ambiguous mathematical formalizations, or unshared or inconsistent code [14, 12, 15, 16].”
Metadata
- Authors
- Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., Nguyen, T. V. N., Glont, M., Hermjakob, H., & Malik‐Sheriff, R. S.
- Affiliations
- Babraham Institute (GB)
- European Bioinformatics Institute (GB)
- Corresponding author
- Hermjakob, Henning
- Malik‐Sheriff, Rahuman S
- ORCID
- Venue
- Molecular Systems Biology
- Published
- 2021-02-01
- Volume
- 17
- Issue
- 2
- Pages
- e9982-e9982
- ISSN
- 1744-4292 (ISSN-L)
- PMC
- PMC7901289
- OpenAlex
- W3132502413
- Cited by
- 131
- References
- 12
- Citations by year
- 2026: 18 · 2025: 21 · 2024: 21 · 2023: 27 · 2022: 27 · 2021: 16
- FWCI
- 19.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- other-oa
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor, 22 d)
- Locations
- Molecular Systems Biology (publishedVersion, other-oa)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Innovative Medicines Initiative (BB/N019482/1, 116030, BB/N019474/1)
- Biotechnology and Biological Sciences Research Council (BB/N019482/1, BB/N019474/1)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Keywords
- reproducibility, systems biology modeling, mathematical model, scientific credibility, model documentation
- MeSH terms
- Humans, Models, Theoretical, Reproducibility of Results, Systems Biology, Data Curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.