Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index; 27 of the 55 sources its claims cite were checked for support.

Adapting modeling and simulation credibility standards to computational systems biology

Open access
Tatka, Lillian T., Smith, Lucian P., Hellerstein, Joseph L., Sauro, Herbert M. · 2023 · Journal of Translational Medicine

Abstract

Computational models are increasingly used in high-impact decision making in science, engineering, and medicine. The National Aeronautics and Space Administration (NASA) uses computational models to perform complex experiments that are otherwise prohibitively expensive or require a microgravity environment. Similarly, the Food and Drug Administration (FDA) and European Medicines Agency (EMA) have began accepting models and simulations as forms of evidence for pharmaceutical and medical device approval. It is crucial that computational models meet a standard of credibility when using them in high-stakes decision making. For this reason, institutes including NASA, the FDA, and the EMA have developed standards to promote and assess the credibility of computational models and simulations. However, due to the breadth of models these institutes assess, these credibility standards are mostly qualitative and avoid making specific recommendations. On the other hand, modeling and simulation in systems biology is a narrower domain and several standards are already in place. As systems biology models increase in complexity and influence, the development of a credibility assessment system is crucial. Here we review existing standards in systems biology, credibility standards in other science, engineering, and medical fields, and propose the development of a credibility standard for systems biology models.

Overview

CoChat Summary

TL;DR

This review surveys existing systems biology modeling standards and credibility frameworks from NASA, FDA, EMA, and ASME, arguing that systems biology’s narrower scope makes it well-suited for a credibility standard. It proposes developing a quantitative or semi-quantitative credibility scoring system for mechanistic systems biology models, potentially informed by automated tools such as MEMOTE and FROG.

Key findings

  • Reproducibility remains a major credibility problem: 49% of published models undergoing BioModels review and curation were not reproducible, primarily due to missing simulation materials, lack of public model/code availability, and insufficient documentation; with extra effort, an additional 12% could be reproduced.
  • Systems biology already has standards for model representation (SBML, CellML, BioPAX), annotation (MIRIAM, SBO, OMEX), simulation and parameter estimation (MIASE, KiSAO, SED-ML, PEtab), and dissemination (FAIR, COMBINE archives), but no widely used standard for model credibility.
  • Credibility standards in other fields are largely qualitative: NASA’s Credibility Assessment Scale has eight factors grouped into development, operations, and management, each scored 0–4; FDA guidance has ten categories and no scoring; EMA’s PBPK guideline requires sensitivity analysis, executable simulation files, and discussion of uncertainty.
  • Automated model-testing tools exist—MEMOTE scores genome-scale metabolic models, and FROG analysis generates reproducible reference datasets—but the paper states that a high MEMOTE score is not itself an assessment of credibility.

Methods

  • Narrative review and proposal; no new experimental data or primary sample are reported.
  • The authors review current systems biology standards, credibility guidelines in systems biology and other fields, and tools for systems biology model testing.
  • The main analysis is a conceptual synthesis arguing that existing standards could support development of a credibility standard for mechanistic systems biology models.

Limitations

  • Stated: no standard for model credibility in systems biology exists; existing credibility standards are mostly qualitative and avoid specific recommendations; FDA guidelines are nonbinding and lack scoring; context-of-use dependence complicates automated credibility assessment.
  • Stated: NASA’s CAS still has a significant subjective component, and different decision makers may assign different credibility to the same model.
  • Evident from the text: this is a review and proposal without empirical validation or a concrete scoring rubric, and no systematic review methodology is described.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s12967-023-04290-5

Cites

70 of 70 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 The Committee on Credible Practice of Modeling and Simulation in Healthcare 10 rules of model credibility
Fig. 1 The Committee on Credible Practice of Modeling and Simulation in Healthcare 10 rules of model credibilityCC BY · PubMed Central
Fig. 2 Categories of the NASA Credibility Assessment Scale (CAS)
Fig. 2 Categories of the NASA Credibility Assessment Scale (CAS)CC BY · PubMed Central

Verification

Claims the paper makes

22 Valid · 3 Not valid · 18 Not checked · 47 Own results and interpretations

18 more claims · 2 with their cited sources checked

Create a free account to see all 90 claims and to verify the 5 references still open.

See all claims — free account
  • Probably not a substantial claim. 44
  • It may state two separate claims as one. 43

Create a free account to see these claims.

Citations

Cites

Show all 69 references

Research works printed but not linked: 2

  1. Klipp Edda. Wolfram Liebermeister. Anselm Helbig: Axel Kowald, and Jörg schaber. standards in computational systems biology; 2007.
  2. Olivier Brett G, Bergmann Frank T. SBML level 3 package: flux balance constraints version 2. J Integr Bioinform. 2018. https://doi.org/10.1515/jib-2017-0082. · DOI · Paper page

Cited by

See all 27

Metadata

Authors
Tatka, L. T., Smith, L. P., Hellerstein, J. L., & Sauro, H. M.
Affiliations
University of Washington (US)
Corresponding author
Tatka, Lillian T.
Venue
Journal of Translational Medicine
Published
2023-07-26
Volume
21
Issue
1
Pages
501-501
ISSN
1479-5876 (ISSN-L)
OpenAlex
W4385295938
Cited by
27
References
69
Citations by year
2026: 11 · 2025: 7 · 2024: 6 · 2023: 3
FWCI
2.65
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Translational Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
  • National Institute of Biomedical Imaging and Bioengineering (P41GM109824)
  • National Science Foundation (1933453)
Topics
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Philosophy of Biology and Scientific Explanation
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
Keywords
computational systems biology, model credibility, modeling and simulation, regulatory standards, FDA, EMA, NASA, model validation, evidence-based decision making
MeSH terms
Computer Simulation, Computational Biology, Systems Biology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Adapting modeling and simulation credibility standards to computational systems biology. https://app.cochat.ai/papers/10.1186/s12967-023-04290-5
The paper
Tatka, L. T., Smith, L. P., Hellerstein, J. L., & Sauro, H. M. (2023). Adapting modeling and simulation credibility standards to computational systems biology. Journal of Translational Medicine. https://doi.org/10.1186/s12967-023-04290-5

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.