The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index; 27 of the 55 sources its claims cite were checked for support.
Adapting modeling and simulation credibility standards to computational systems biology
Open accessAbstract
Computational models are increasingly used in high-impact decision making in science, engineering, and medicine. The National Aeronautics and Space Administration (NASA) uses computational models to perform complex experiments that are otherwise prohibitively expensive or require a microgravity environment. Similarly, the Food and Drug Administration (FDA) and European Medicines Agency (EMA) have began accepting models and simulations as forms of evidence for pharmaceutical and medical device approval. It is crucial that computational models meet a standard of credibility when using them in high-stakes decision making. For this reason, institutes including NASA, the FDA, and the EMA have developed standards to promote and assess the credibility of computational models and simulations. However, due to the breadth of models these institutes assess, these credibility standards are mostly qualitative and avoid making specific recommendations. On the other hand, modeling and simulation in systems biology is a narrower domain and several standards are already in place. As systems biology models increase in complexity and influence, the development of a credibility assessment system is crucial. Here we review existing standards in systems biology, credibility standards in other science, engineering, and medical fields, and propose the development of a credibility standard for systems biology models.
Overview
CoChat Summary
TL;DR
This review surveys existing systems biology modeling standards and credibility frameworks from NASA, FDA, EMA, and ASME, arguing that systems biology’s narrower scope makes it well-suited for a credibility standard. It proposes developing a quantitative or semi-quantitative credibility scoring system for mechanistic systems biology models, potentially informed by automated tools such as MEMOTE and FROG.
Key findings
- Reproducibility remains a major credibility problem: 49% of published models undergoing BioModels review and curation were not reproducible, primarily due to missing simulation materials, lack of public model/code availability, and insufficient documentation; with extra effort, an additional 12% could be reproduced.
- Systems biology already has standards for model representation (SBML, CellML, BioPAX), annotation (MIRIAM, SBO, OMEX), simulation and parameter estimation (MIASE, KiSAO, SED-ML, PEtab), and dissemination (FAIR, COMBINE archives), but no widely used standard for model credibility.
- Credibility standards in other fields are largely qualitative: NASA’s Credibility Assessment Scale has eight factors grouped into development, operations, and management, each scored 0–4; FDA guidance has ten categories and no scoring; EMA’s PBPK guideline requires sensitivity analysis, executable simulation files, and discussion of uncertainty.
- Automated model-testing tools exist—MEMOTE scores genome-scale metabolic models, and FROG analysis generates reproducible reference datasets—but the paper states that a high MEMOTE score is not itself an assessment of credibility.
Methods
- Narrative review and proposal; no new experimental data or primary sample are reported.
- The authors review current systems biology standards, credibility guidelines in systems biology and other fields, and tools for systems biology model testing.
- The main analysis is a conceptual synthesis arguing that existing standards could support development of a credibility standard for mechanistic systems biology models.
Limitations
- Stated: no standard for model credibility in systems biology exists; existing credibility standards are mostly qualitative and avoid specific recommendations; FDA guidelines are nonbinding and lack scoring; context-of-use dependence complicates automated credibility assessment.
- Stated: NASA’s CAS still has a significant subjective component, and different decision makers may assign different credibility to the same model.
- Evident from the text: this is a review and proposal without empirical validation or a concrete scoring rubric, and no systematic review methodology is described.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
22 Valid · 3 Not valid · 18 Not checked · 47 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“With some extra effort, an additional 12% of the published models could be reproduced.”
§ Adapting modeling and simulation credibility standards to computational systems biology > Introduction
What the sources say
- Supportedcites [7] Krishna Tiwari, Sarubini Kananathan, Roberts Matthew G, Meyer Johannes P, Sharif Shohan Mohammad Umer, Ashley Xavier, Matthieu Maire, Ahmad Zyoud, Jinghao Men, Szeyi Ng, Nguyen Tung VN, Mihai Glont, Henning Hermjakob, Malik-Sheriff Rahuman S. Reproducibility in systems biology modelling. Mol Syst Biol. 2021;17(2): e9982. source
The section “12% of the models could be reproduced with further effort” states that about 12% of all models were reproduced through empirical corrections or author support. Figure 1B repeats this result. The claim’s “additional” distinguishes these models from those reproduced directly, as the paper does. (LLM, conf=0.95)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Many modelers reproduce simulations by reading the simulation description in the corresponding publication [46].”
§ Adapting modeling and simulation credibility standards to computational systems biology > Current standards in systems biology > Simulation and parameter estimation
What the sources say
- Supportedcites [46] ...Waltemath Dagmar, Adams Richard, Beard Daniel A, Bergmann Frank T, Bhalla Upinder S, Britten Randall, Chelliah Vijayalakshmi, Cooling Michael T, Cooper Jonathan, Crampin Edmund J, Garny Alan, Hoops Stefan, Hucka Michael, Hunter Peter, Klipp Edda, Laibe Camille, Miller Andrew K, Moraru Ion, Nickerson David, Nielsen Poul, Nikolski Macha, Sahle Sven, Sauro Herbert M, Schmidt Henning, Snoep Jacky L, Tolle Dominic, Wolkenhauer Olaf, Le Novère Nicolas. Minimum information about a simulation experiment (MIASE). PLoS Comput Biol. 2011;7(4): e1001122. source
In “Needs for a Standard Description of Simulations Experiments,” the paper says that users typically read the simulation description in the corresponding publication when trying existing simulation protocols. This directly supports the citing sentence’s paraphrase. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“For this reason, systems biology model representation has become standardized using XML-based languages SBML [10, 13], CellML [14], and BioPAX [15].”
§ Adapting modeling and simulation credibility standards to computational systems biology > Current standards in systems biology > Model representation
What the sources say
- Not matchedcites [10] Klipp Edda. Wolfram Liebermeister. Anselm Helbig: Axel Kowald, and Jörg schaber. standards in computational systems biology; 2007.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [13] Machado Daniel, Costa Rafael S, Rocha Miguel, Ferreira Eugénio C, Tidor Bruce. Modeling formalisms in systems biology. AMB Express. 2011;1:45. source
The citation to this paper is attached to SBML, not to CellML or BioPAX. In the Standardization section, the paper calls SBML the de facto standard of the systems biology community and describes it as an XML-based language for model representation. (LLM, conf=0.94)
- Not checked — no full text of the sourcecites [14] Cuellar Autumn A, Lloyd Catherine M, Nielsen Poul F, Bullivant David P, Nickerson David P, Hunter Peter J. An overview of CellML 1.1, a biological model description language. Simulation. 2003;79(12):740–7. source
- Supportedcites [15] ...Demir Emek, Cary Michael P, Paley Suzanne, Fukuda Ken, Lemer Christian, Vastrik Imre, Guanming Wu, D’Eustachio Peter, Schaefer Carl, Luciano Joanne, Schacherer Frank, Martinez-Flores Irma, Zhenjun Hu, Jimenez-Jacinto Veronica, Joshi-Tope Geeta, Kandasamy Kumaran, Lopez-Fuentes Alejandra C, Mi Huaiyu, Pichler Elgar, Rodchenkov Igor, Splendiani Andrea, Tkachev Sasha, Zucker Jeremy, Gopinath Gopal, Rajasimha Harsha, Ramakrishnan Ranjani, Shah Imran, Syed Mustafa, Anwar Nadia, Babur Ozgun, Blinov Michael, Brauner Erik, Corwin Dan, Donaldson Sylva, Gibbons Frank, Goldberg Robert, Hornbeck Peter, source
The Implementation section describes BioPAX as a standard language for representing biological pathways. The Language evolution section states that its OWL-based ontology and data can be stored in an XML-based format. The BioPAX citation supports the BioPAX portion of the sentence; SBML and CellML have separate citations. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Due to the increasing importance of computational models in scientific discovery, the National Aeronautics and Space Administration (NASA), the FDA, and other regulatory bodies have developed standards to assess the credibility of models [4, 8, 9].”
§ Adapting modeling and simulation credibility standards to computational systems biology > Introduction
What the sources say
- Not matchedcites [4] Assessing the credibility of computational modeling and simulation in medical device submissions: Draft guidance for industry and food and drug administration staff. U.S. Food and Drug Administration, 2021.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [8] Babula Maria, Bertch William, Green Lawrence, Hale Joseph, Mosier Gary, Steele Martin, Woods Jody. NASA Standard for Models and Simulations: Credibility Assessment Scale. In 47th AIAA Aerospace Sciences Meeting including The New Horizons Forum and Aerospace Exposition, Orlando, Florida. American Institute of Aeronautics and Astronautics.2007.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [9] Shepard T, Scott G, Cole S, Nordmark A, Bouzom F. Physiologically based models in regulatory submissions: Output from the ABPI/MHRA forum on physiologically based modeling and simulation. CPT Pharmacometrics Syst Pharmacol. 2015;4(4):221–5. sourceExperimental — accuracy not yet measured
The European Regulatory Viewpoint (page unavailable) reports a published European PBPK concept paper and plans for a guideline on model qualification and reporting. It mentions an earlier FDA workshop, but does not say the FDA developed credibility standards; the Discussion Topics section says PBPK system-model qualification standards were not yet agreed. Our copy does not mention NASA. (LLM, conf=0.87)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“It was recently discovered that 49% of published models undergoing the review and curation process for the BioModels [6] database were not reproducible primarily due to missing materials necessary for simulation, the availability of the model and code in public databases, and lack of documentation [7].”
§ Adapting modeling and simulation credibility standards to computational systems biology > Introduction
What the sources say
- Partly supportedcites [7] Krishna Tiwari, Sarubini Kananathan, Roberts Matthew G, Meyer Johannes P, Sharif Shohan Mohammad Umer, Ashley Xavier, Matthieu Maire, Ahmad Zyoud, Jinghao Men, Szeyi Ng, Nguyen Tung VN, Mihai Glont, Henning Hermjakob, Malik-Sheriff Rahuman S. Reproducibility in systems biology modelling. Mol Syst Biol. 2021;17(2): e9982. sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
18 more claims · 2 with their cited sources checked
Create a free account to see all 90 claims and to verify the 5 references still open.
- Probably not a substantial claim. 44
- It may state two separate claims as one. 43
Citations
Cites
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
UniProt: the Universal Protein knowledgebase
Apweiler, R. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh131 · Cited by 7,498
Hucka, M., Finney, A., Sauro, H. M., Bolouri, H., Doyle, J. C., Kitano, H., Arkin, A. P., Bornstein, B. J., Bray, D., Cornish-Bowden, A., et al. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btg015 · Cited by 3,060
Self-citationChEBI in 2016: Improved services and an expanding collection of metabolites
Hastings, J., Owen, G., Dekker, A., Ennis, M., Kale, N., Muthukrishnan, V., Turner, S., Swainston, N., Mendes, P., & Steinbeck, C. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkv1031 · Cited by 1,083
The systems biology graphical notation.
Novère, N. L., Hucka, M., Mi, H., Moodie, S., Schreiber, F., Sorokin, A., Demir, E., Wegner, K., Aladjem, M. I., Wimalaratne, S. M., et al. (2009) · Nature Biotechnology · doi:10.1038/nbt.1558 · Cited by 903
Self-citationUniform Resource Identifier (URI): Generic Syntax
Berners-Lee, T., Fielding, R., & Masinter, L. (2005) · doi:10.17487/rfc3986 · Cited by 815
Le Novere, N. (2006) · Nucleic Acids Research · doi:10.1093/nar/gkj092 · Cited by 762
The BioPAX community standard for pathway data sharing
Demir, E., Cary, M. P., Paley, S., Fukuda, K., Lemer, C., Vastrik, I., Wu, G., D'Eustachio, P., Schaefer, C., Luciano, J., et al. (2010) · Nature Biotechnology · doi:10.1038/nbt.1666 · Cited by 745
The Semantic Web: the roles of XML and RDF
Decker, S., Melnik, S., van Harmelen, F., Fensel, D., Klein, M., Broekstra, J., Erdmann, M., & Horrocks, I. (2000) · IEEE Internet Computing · doi:10.1109/4236.877487 · Cited by 743
Drug–target interaction prediction: databases, web servers and computational models
Chen, X., Yan, C. C., Zhang, X., Zhang, X., Dai, F., Yin, J., & Zhang, Y. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbv066 · Cited by 658
Research works printed but not linked: 2
- Klipp Edda. Wolfram Liebermeister. Anselm Helbig: Axel Kowald, and Jörg schaber. standards in computational systems biology; 2007.
- Olivier Brett G, Bergmann Frank T. SBML level 3 package: flux balance constraints version 2. J Integr Bioinform. 2018. https://doi.org/10.1515/jib-2017-0082. · DOI · Paper page
Cited by
Microbiome modeling: a beginner's guide
Lange, E., Kranert, L., Krüger, J., Benndorf, D., & Heyer, R. (2024) · Frontiers in Microbiology · doi:10.3389/fmicb.2024.1368377 · Cited by 21
Multi-scale models of whole cells: progress and challenges
Georgouli, K., Yeom, J.-S., Blake, R. C., & Navid, A. (2023) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2023.1260507 · Cited by 22
“Developments of tools like Memote (Lieven et al., 2020) for standardizing the GEMs and FROG ensemble of analyses for ensuring reproducibility of published models (Tatka et al., 2023) have signi fi cantly increased con fi dence in the quality of models that will be incorporated in future WCMs.”
MethodTolk, A. (2023) · SIMULATION · doi:10.1177/00375497231216471 · Cited by 21
“41 There are more application domains than defense and healthcare, but these two are the most active ones and supported by numerous research projects.”
Cancer drug resistance as learning of signaling networks
Keresztes, D., Kerestély, M., Szarka, L., Kovács, B. M., Schulc, K., Veres, D. V., & Csermely, P. (2025) · Biomedicine & Pharmacotherapy · doi:10.1016/j.biopha.2025.117880 · Cited by 20
Michael, C. T., Almohri, S. A., Linderman, J. J., & Kirschner, D. E. (2024) · Frontiers in Systems Biology · doi:10.3389/fsysb.2023.1283341 · Cited by 18
Metadata
- Authors
- Tatka, L. T., Smith, L. P., Hellerstein, J. L., & Sauro, H. M.
- Affiliations
- University of Washington (US)
- Corresponding author
- Tatka, Lillian T.
- Venue
- Journal of Translational Medicine
- Published
- 2023-07-26
- Volume
- 21
- Issue
- 1
- Pages
- 501-501
- ISSN
- 1479-5876 (ISSN-L)
- PMC
- PMC10369698
- OpenAlex
- W4385295938
- Cited by
- 27
- References
- 69
- Citations by year
- 2026: 11 · 2025: 7 · 2024: 6 · 2023: 3
- FWCI
- 2.65
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Journal of Translational Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Institute of Biomedical Imaging and Bioengineering (P41GM109824)
- National Science Foundation (1933453)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Philosophy of Biology and Scientific Explanation
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Keywords
- computational systems biology, model credibility, modeling and simulation, regulatory standards, FDA, EMA, NASA, model validation, evidence-based decision making
- MeSH terms
- Computer Simulation, Computational Biology, Systems Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Adapting modeling and simulation credibility standards to computational systems biology. https://app.cochat.ai/papers/10.1186/s12967-023-04290-5
- The paper
- Tatka, L. T., Smith, L. P., Hellerstein, J. L., & Sauro, H. M. (2023). Adapting modeling and simulation credibility standards to computational systems biology. Journal of Translational Medicine. https://doi.org/10.1186/s12967-023-04290-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

