The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 3 of the 4 sources its claims cite were checked for support.
Reproducibility in systems biology modelling
Open accessPublished version of a preprint
Abstract
Reproducibility of scientific results is a key element of science and credibility. The lack of reproducibility across many scientific fields has emerged as an important concern. In this piece, we assess mathematical model reproducibility and propose a scorecard for improving reproducibility in this field.
Overview
CoChat Summary
TL;DR
A systematic BioModels curation study of 455 published ODE models found that 51% directly reproduced at least one main simulation figure, while 49% did not; further effort raised reproducibility to 63%. The authors propose an 8-point reproducibility scorecard for modellers, reviewers, and journal editors.
Key findings
- Of 455 kinetic models, 233 (51%) were directly reproducible; about 49% were not reproducible due to incorrect or missing information in the manuscript. Among 66 models submitted to BioModels in SBML by authors, only 37 could be reproduced directly.
- About 12% of models were reproduced with further effort: 40 models (9%) with manual empirical corrections and 13 models (3%) with author support, bringing total reproducibility to about 63%. The authors contacted corresponding authors for 90 models; fewer than a third (27) responded, and about 70% did not respond.
- Of the 169 non-reproducible models (37%), 99 had identified reasons: missing parameter values (n = 52), missing initial conditions (n = 44), and inconsistency in model structure (n = 36), including combinations; in 70 models the reason was unclear.
- In a randomly selected subset of 110 models, a score ≥ 4 on the 8-point scorecard was significantly associated with reproducibility: χ²(1, N = 110) = 10.0733, P = 0.0015, odds ratio = 4.62 (95% CI: 1.71, 12.44). Questions Q2–Q8 were significantly associated with reproducibility (P < 0.05); Q1 could not be tested because all 110 models scored “yes.”
Methods
- The study systematically analysed 455 published ODE models from 152 journals, representing over 20% of all public literature-based models in BioModels. Models were not randomly sampled but selected based on BioModels curation priorities, including funding, collaborations, curators’ interest, and direct submissions.
- Mathematical equations for 389 models were manually encoded in SBML using COPASI from the original manuscripts; 66 author-submitted SBML models were cross-checked. Simulations were predominantly performed in COPASI, or in other software such as SimBiology, libSBMLsim, and Mathematica when COPASI was used originally.
- A model was considered reproducible if it reproduced at least one main simulation figure using software different from the original, with minor deviations acceptable if they did not affect the scientific conclusion. Models were classified as directly reproducible, reproduced with manual corrections, reproduced with author support, or non-reproducible, and reasons for failure were recorded. The scorecard was tested on 110 randomly selected models using chi-square tests.
Limitations
- The paper states the sample was not randomized and was selected based on BioModels curation priorities, though it covered models from 152 journals. It also states the focus was only on ODE models, one type of kinetic model; other kinetic models are likely affected to the same extent or more, and non-reproducible models were less likely to be selected for curation, so the 49% non-reproducibility rate is likely an underestimate.
- The paper states the reproducibility criterion required reproducing at least one figure, not all relevant figures, as a compromise between reliability and effort. It also notes possible curator error but argues this is unlikely to alter conclusions because curators consult as a team and non-reproducibility was similar among author-submitted SBML models (29 of 66, 44%).
- The paper states author contact was not always feasible due to authors’ change of institutions, change of field, leaving academia, or death, and only 27 of 90 contacted authors responded. It also notes non-reproducible models are not explicitly labelled in BioModels because failure could be due to curator error and to avoid discouraging authors from sharing models.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
3 Valid · 0 Not valid · 1 Not checked · 44 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, mathematical models from a number of published articles were shown not to reproduce the simulation results described in the article (Mendes, 2018).”
§ Reproducibility in systems biology modelling > Abstract
What the sources say
- Partly supportedcites Mendes, 2018 sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says many computational articles are hard or impossible to reproduce, and Some recommendations (page unavailable) reports that only a minority of roughly 400 surveyed biomodel articles described their work sufficiently for reproduction. These passages support a reproducibility problem, but do not say that the mathematical models themselves were tested and shown to produce results different from those reported. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similar to the curation service in BioModels, JWSOnline and the Physiome repository through the Centre for Reproducible BioMedical modelling (Papin et al, 2020) provide expert curation services to validate published reports.”
§ Reproducibility in systems biology modelling > Reproducibility scorecard > Modelling guidelines/community efforts
What the sources say
- Partly supportedcites Papin et al, 2020 sourceExperimental — accuracy not yet measured
In the main text (page not specified), Papin et al. describe a partnership with the Center for Reproducible Biomedical Modeling for peer review of computational-model reproducibility. Our copy does not mention JWSOnline or the Physiome repository, or establish that either provides expert curation services to validate published reports. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“BioModels, being the largest repository of open, curated, well‐annotated and findable models, contributes significantly to reproducibility (Mendes, 2018).”
§ Reproducibility in systems biology modelling > Reproducibility scorecard > Modelling guidelines/community efforts
What the sources say
- Partly supportedcites Mendes, 2018 sourceExperimental — accuracy not yet measured
In “Publication of electronic materials” (page unavailable), Mendes calls BioModels the most widely used model repository and says its curators verify published results, making submission a major check on reproducibility. The section also says public repositories help materials remain findable. It does not establish that BioModels is the largest repository or that its models are well annotated. (LLM, conf=0.85)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Forty models (9%) were successfully reproduced with manual empirical correction of the inaccurate reporting in the manuscript.”
§ Reproducibility in systems biology modelling > 12% of the models could be reproduced with further effort
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“About half of the models of authors who responded (13 models, or 3% of the total number of models) were subsequently reproduced.”
§ Reproducibility in systems biology modelling > 12% of the models could be reproduced with further effort
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 48 claims and every source verdict.
- It may state two separate claims as one. 12
- The quote only partly states this claim. 1
- Probably not a substantial claim. 9
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It was recently discovered that 49% of published models undergoing the review and curation process for the BioModels [6] database were not reproducible primarily due to missing materials necessary for simulation, the availability of the model and code in public databases, and lack of documentation [7].”
From Adapting modeling and simulation credibility standards to computational systems biology (2023) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“With some extra effort, an additional 12% of the published models could be reproduced.”
From Adapting modeling and simulation credibility standards to computational systems biology (2023) · cites this paper as [7]
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentχ²(1) = 10.0733reported p = .0015 · recomputed p = .002
“X2 (1, N = 110) = 10.0733, p = .0015”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Self-check results appear here once an independent check has confirmed them.
Citations
Cites
Gene Ontology: tool for the unification of biology
Ashburner, M., Ball, C. A., Blake, J. A., Botstein, D., Butler, H., Cherry, J. M., Davis, A. P., Dolinski, K., Dwight, S. S., Eppig, J. T., et al. (2000) · Nature Genetics · doi:10.1038/75556 · Cited by 43,206
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
UniProt: a worldwide hub of protein knowledge
The UniProt Consortium (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1049 · Cited by 7,698
The Gene Ontology Resource: 20 years and still GOing strong
The Gene Ontology Consortium (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1055 · Cited by 4,747
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
COPASI—a COmplex PAthway SImulator
Hoops, S., Sahle, S., Gauges, R., Lee, C., Pahle, J., Simus, N., Singhal, M., Xu, L., Mendes, P., & Kummer, U. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl485 · Cited by 2,618
The reactome pathway knowledgebase
Jassal, B., Matthews, L., Viteri, G., Gong, C., Lorente, P., Fabregat, A., Sidiropoulos, K., Cook, J., Gillespie, M., Haw, R., et al. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz1031 · Cited by 2,159
Self-citationFederhen, S. (2012) · Nucleic Acids Research · doi:10.1093/nar/gkr1178 · Cited by 1,436
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · cited 2×
“Although the lack of reproducibility has been discussed within the community (Mendes, 2018; Papin et al , 2020), it was not expected to be this adverse.”
Cited by
Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers
Wratten, L., Wilm, A., & Göke, J. (2021) · Nature Methods · doi:10.1038/s41592-021-01254-9 · Cited by 225
Musuamba, F. T., Skottheim Rusten, I., Lesage, R., Russo, G., Bursi, R., Emili, L., Wangorsch, G., Manolis, E., Karlsson, K. E., Kulesza, A., et al. (2021) · CPT: Pharmacometrics & Systems Pharmacology · doi:10.1002/psp4.12669 · Cited by 164
To Sobol or not to Sobol? The effects of sampling schemes in systems biology applications
Renardy, M., Joslyn, L. R., Millar, J. A., & Kirschner, D. E. (2021) · Mathematical Biosciences · doi:10.1016/j.mbs.2021.108593 · Cited by 182
The European Bioinformatics Institute (EMBL-EBI) in 2021
Cantelli, G., Bateman, A. G., Brooksbank, C., Petrov, A. I., Malik‐Sheriff, R. S., Ide‐Smith, M., Hermjakob, H., Flicek, P., Apweiler, R., Birney, E., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab1127 · Cited by 101
Self-citationImmune digital twins for complex human pathologies: applications, limitations, and challenges
Niarakis, A., Laubenbacher, R., An, G. C., Ilan, Y., Fisher, J., Flobak, Å., Reiche, K., Martínez, M. R., Geris, L., Ladeira, L., et al. (2024) · npj Systems Biology and Applications · doi:10.1038/s41540-024-00450-5 · Cited by 127
Self-citation
Metadata
- Authors
- Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., Nguyen, T. V. N., Glont, M., Hermjakob, H., & Malik‐Sheriff, R. S.
- Affiliations
- Babraham Institute (GB)
- European Bioinformatics Institute (GB)
- Corresponding author
- Hermjakob, Henning
- Malik‐Sheriff, Rahuman S
- ORCID
- Venue
- Molecular Systems Biology
- Published
- 2021-02-01
- Volume
- 17
- Issue
- 2
- Pages
- e9982-e9982
- ISSN
- 1744-4292 (ISSN-L)
- PMC
- PMC7901289
- OpenAlex
- W3132502413
- Cited by
- 131
- References
- 12
- Citations by year
- 2026: 18 · 2025: 21 · 2024: 21 · 2023: 27 · 2022: 27 · 2021: 16
- FWCI
- 19.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- other-oa
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor, 22 d)
- Locations
- Molecular Systems Biology (publishedVersion, other-oa)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Innovative Medicines Initiative (BB/N019482/1, 116030, BB/N019474/1)
- Biotechnology and Biological Sciences Research Council (BB/N019482/1, BB/N019474/1)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Keywords
- reproducibility, systems biology modeling, mathematical model, scientific credibility, model documentation
- MeSH terms
- Humans, Models, Theoretical, Reproducibility of Results, Systems Biology, Data Curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Reproducibility in systems biology modelling. https://app.cochat.ai/papers/10.15252/msb.20209982
- The paper
- Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., Nguyen, T. V. N., Glont, M., Hermjakob, H., & Malik‐Sheriff, R. S. (2021). Reproducibility in systems biology modelling. Molecular Systems Biology. https://doi.org/10.15252/msb.20209982
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.