Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 3 of the 4 sources its claims cite were checked for support.

Reproducibility in systems biology modelling

Open access

Published version of a preprint

Abstract

Reproducibility of scientific results is a key element of science and credibility. The lack of reproducibility across many scientific fields has emerged as an important concern. In this piece, we assess mathematical model reproducibility and propose a scorecard for improving reproducibility in this field.

Overview

CoChat Summary

TL;DR

A systematic BioModels curation study of 455 published ODE models found that 51% directly reproduced at least one main simulation figure, while 49% did not; further effort raised reproducibility to 63%. The authors propose an 8-point reproducibility scorecard for modellers, reviewers, and journal editors.

Key findings

  • Of 455 kinetic models, 233 (51%) were directly reproducible; about 49% were not reproducible due to incorrect or missing information in the manuscript. Among 66 models submitted to BioModels in SBML by authors, only 37 could be reproduced directly.
  • About 12% of models were reproduced with further effort: 40 models (9%) with manual empirical corrections and 13 models (3%) with author support, bringing total reproducibility to about 63%. The authors contacted corresponding authors for 90 models; fewer than a third (27) responded, and about 70% did not respond.
  • Of the 169 non-reproducible models (37%), 99 had identified reasons: missing parameter values (n = 52), missing initial conditions (n = 44), and inconsistency in model structure (n = 36), including combinations; in 70 models the reason was unclear.
  • In a randomly selected subset of 110 models, a score ≥ 4 on the 8-point scorecard was significantly associated with reproducibility: χ²(1, N = 110) = 10.0733, P = 0.0015, odds ratio = 4.62 (95% CI: 1.71, 12.44). Questions Q2–Q8 were significantly associated with reproducibility (P < 0.05); Q1 could not be tested because all 110 models scored “yes.”

Methods

  • The study systematically analysed 455 published ODE models from 152 journals, representing over 20% of all public literature-based models in BioModels. Models were not randomly sampled but selected based on BioModels curation priorities, including funding, collaborations, curators’ interest, and direct submissions.
  • Mathematical equations for 389 models were manually encoded in SBML using COPASI from the original manuscripts; 66 author-submitted SBML models were cross-checked. Simulations were predominantly performed in COPASI, or in other software such as SimBiology, libSBMLsim, and Mathematica when COPASI was used originally.
  • A model was considered reproducible if it reproduced at least one main simulation figure using software different from the original, with minor deviations acceptable if they did not affect the scientific conclusion. Models were classified as directly reproducible, reproduced with manual corrections, reproduced with author support, or non-reproducible, and reasons for failure were recorded. The scorecard was tested on 110 randomly selected models using chi-square tests.

Limitations

  • The paper states the sample was not randomized and was selected based on BioModels curation priorities, though it covered models from 152 journals. It also states the focus was only on ODE models, one type of kinetic model; other kinetic models are likely affected to the same extent or more, and non-reproducible models were less likely to be selected for curation, so the 49% non-reproducibility rate is likely an underestimate.
  • The paper states the reproducibility criterion required reproducing at least one figure, not all relevant figures, as a compromise between reliability and effort. It also notes possible curator error but argues this is unlikely to alter conclusions because curators consult as a team and non-reproducibility was similar among author-submitted SBML models (29 of 66, 44%).
  • The paper states author contact was not always feasible due to authors’ change of institutions, change of field, leaving academia, or death, and only 27 of 90 contacted authors responded. It also notes non-reproducible models are not explicitly labelled in BioModels because failure could be due to curator error and to avoid discouraging authors from sharing models.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.15252/msb.20209982

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Reproducibility of systems biology models (A) About half of the published systems biology models could not be directly reproduced. (B) About 12% of the models could be reproduced with empirical corrections or author support. (C) The year of publication of the models analysed in this study. (D) Distribution of model size (number of entities/species) (top) and percentage of reproducible (directly reproducible and reproduced with further efforts) and non‐reproducible models (bottom) at specified bins of model size. (E) Distribution of directly reproducible, reproduced with further efforts (empirical correction and author support) and non‐reproducible models across various journals with more than two models in our work.
Figure 1 Reproducibility of systems biology models (A) About half of the published systems biology models could not be directly reproduced. (B) About 12% of the models could be reproduced with empirical corrections or author support. (C) The year of publication of the models analysed in this study. (D) Distribution of model size (number of entities/species) (top) and percentage of reproducible (directly reproducible and reproduced with further efforts) and non‐reproducible models (bottom) at specified bins of model size. (E) Distribution of directly reproducible, reproduced with further efforts (empirical correction and author support) and non‐reproducible models across various journals with more than two models in our work.CC BY · PubMed Central
Figure 2 The 8‐point reproducibility scorecard is an indicator of model reproducibility (A) Distribution of total reproducibility score across models*. (B) Percentage of non‐reproducible and reproducible models for each total score. (C) Distribution of points scored by models* from each question in the scorecard. Each “yes” answer to a question in the scorecard will gain a score of 1. (D) Comparison of points scored by non‐reproducible and reproducible models from each question. *110 models (45 Not reproducible and 65 reproducible**) from the 455 models were scored using the scorecard. **Reproducible models include both directly reproducible models (n = 46) and those reproduced with further efforts (n = 19). Q1 – Q8 are questions representing 8 points in the reproducibility scorecard.
Figure 2 The 8‐point reproducibility scorecard is an indicator of model reproducibility (A) Distribution of total reproducibility score across models*. (B) Percentage of non‐reproducible and reproducible models for each total score. (C) Distribution of points scored by models* from each question in the scorecard. Each “yes” answer to a question in the scorecard will gain a score of 1. (D) Comparison of points scored by non‐reproducible and reproducible models from each question. *110 models (45 Not reproducible and 65 reproducible**) from the 455 models were scored using the scorecard. **Reproducible models include both directly reproducible models (n = 46) and those reproduced with further efforts (n = 19). Q1 – Q8 are questions representing 8 points in the reproducibility scorecard.CC BY · PubMed Central

Verification

Claims the paper makes

3 Valid · 0 Not valid · 1 Not checked · 44 Own results and interpretations

25 more claims

Create a free account to see all 48 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 12
  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 9

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentχ²(1) = 10.0733reported p = .0015 · recomputed p = .002

    “X2 (1, N = 110) = 10.0733, p = .0015”

1 of 1

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Citations

Cites

Show all 12 references

Cited by

See all 131

Metadata

Authors
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., Nguyen, T. V. N., Glont, M., Hermjakob, H., & Malik‐Sheriff, R. S.
Affiliations
  • Babraham Institute (GB)
  • European Bioinformatics Institute (GB)
Corresponding author
  • Hermjakob, Henning
  • Malik‐Sheriff, Rahuman S
Venue
Molecular Systems Biology
Published
2021-02-01
Volume
17
Issue
2
Pages
e9982-e9982
ISSN
1744-4292 (ISSN-L)
OpenAlex
W3132502413
Cited by
131
References
12
Citations by year
2026: 18 · 2025: 21 · 2024: 21 · 2023: 27 · 2022: 27 · 2021: 16
FWCI
19.44
Type
journal article
Language
en
Open access
Licence
Locations
  • Molecular Systems Biology (publishedVersion, other-oa)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Innovative Medicines Initiative (BB/N019482/1, 116030, BB/N019474/1)
  • Biotechnology and Biological Sciences Research Council (BB/N019482/1, BB/N019474/1)
Topics
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
Keywords
reproducibility, systems biology modeling, mathematical model, scientific credibility, model documentation
MeSH terms
Humans, Models, Theoretical, Reproducibility of Results, Systems Biology, Data Curation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Reproducibility in systems biology modelling. https://app.cochat.ai/papers/10.15252/msb.20209982
The paper
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., Nguyen, T. V. N., Glont, M., Hermjakob, H., & Malik‐Sheriff, R. S. (2021). Reproducibility in systems biology modelling. Molecular Systems Biology. https://doi.org/10.15252/msb.20209982

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.