The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index; 61 of the 99 sources its claims cite were checked for support.
Credibility assessment of in silico clinical trials for medical devices
Open accessAbstract
In silico clinical trials (ISCTs) are an emerging method in modeling and simulation where medical interventions are evaluated using computational models of patients. ISCTs have the potential to provide cost-effective, time-efficient, and ethically favorable alternatives for evaluating the safety and effectiveness of medical devices. However, ensuring the credibility of ISCT results is a significant challenge. This paper aims to identify unique considerations for assessing the credibility of ISCTs and proposes an ISCT credibility assessment workflow based on recently published model assessment frameworks. First, we review various ISCTs described in the literature, carefully selected to showcase the range of methodological options available. These studies cover a wide variety of devices, reasons for conducting ISCTs, patient model generation approaches including subject-specific versus ‘synthetic’ virtual patients, complexity levels of devices and patient models, incorporation of clinician or clinical outcome models, and methods for integrating ISCT results with real-world clinical trials. We next discuss how verification, validation, and uncertainty quantification apply to ISCTs, considering the range of ISCT approaches identified. Based on our analysis, we then present a hierarchical workflow for assessing ISCT credibility, using a general credibility assessment framework recently published by the FDA’s Center for Devices and Radiological Health. Overall, this work aims to promote standardization in ISCTs and contribute to the wider adoption and acceptance of ISCTs as a reliable tool for evaluating medical devices.
Overview
CoChat Summary
TL;DR
This paper reviews eight medical-device in silico clinical trials (ISCTs) and proposes a hierarchical workflow for assessing ISCT credibility based on FDA guidance and ASME V&V40. It argues that ISCT credibility requires sub-model-level verification, validation, and uncertainty quantification, plus population-level validation of the virtual cohort.
Key findings
- ISCTs are composed of multiple sub-models—baseline patient model, device model, virtual cohort, and optionally clinical outcome, clinician, emulator, and other system models—so credibility assessment should be hierarchical rather than only top-level.
- The reviewed examples span subject-specific, synthetic, and mixed virtual cohorts: MR-induced heating used 15 subject-specific models from 12 subjects; VICTRE used 2,986 synthetic patients; FD-PASS used 164 virtual patients from 82 anatomies; the artificial pancreas example used 300 synthetic patients; whole-heart CRT models used 24 subject-specific models altered to heart failure; automata-based ICD models used >11,000 virtual patients; and Micra involved 50 million simulations total.
- Validation can occur at sub-model level (device, patient, coupled device-patient, clinical outcome, clinician) and at cohort level, either individual-level with paired data or population-level without paired data; population-level validation alone risks “right answer for the wrong reason.”
- The proposed workflow aligns with FDA credibility categories and adds credibility factors/gradations for population-level validation of the virtual cohort, including clinical dataset size, subject range, output comparison rigor, and agreement.
Methods
- Narrative review of eight selected ISCTs for medical devices, chosen to showcase methodological range rather than to comprehensively survey all ISCTs.
- Qualitative synthesis of how verification, validation, and uncertainty quantification apply to ISCTs, including sub-model and cohort validation strategies.
- Proposal of a hierarchical ISCT credibility assessment workflow based on FDA guidance “Assessing the Credibility of Computational Modeling and Simulation in Medical Device Submissions” and ASME V&V40-2018, illustrated with a hypothetical whole-heart/pacemaker-lead ISCT.
Limitations
- The paper states the workflow may be perceived as overly burdensome, may require case-specific adjustments, and such adjustments could introduce subjectivity.
- The paper states its scope is first-principles models; data-driven models are not harmonized with first-principles credibility assessment, and ISCT execution is outside scope.
- The paper states there is no widely accepted community definition of an ISCT, and the literature review is not comprehensive; the proposed workflow also assumes specific features such as a physics-based device model, first-principles patient model, no emulator or explicit clinician model, synthetic cohort, and no Bayesian ISCT/CT integration.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
45 Valid · 9 Not valid · 24 Not checked · 105 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“The flow diverter performance assessment (FD-PASS) study [16] was an in silico clinical trial that predicted post-treatment flow reduction in 164 virtual patients using computational fluid dynamics (CFD).”
§ Credibility assessment of in silico clinical trials for medical devices > 2. In silico clinical trials for medical devices > 2.3. Example ISCTs for medical devices
What the sources say
- Supportedcites [16] A Sarrami-Foroushani, 2021, In-silico trial of intracranial flow diverters replicates and expands insights from conventional clinical trials, Nature communications source
The Introduction’s Fig. 1 workflow describes 82 virtual anatomies modeled under normotensive and hypertensive conditions (N = 164), with the 3-D Navier–Stokes equations solved to measure post-treatment aneurysm mean velocity reduction. The Introduction also identifies FD-PASS as an in-silico trial with post-treatment velocity reduction as its primary endpoint. (LLM, conf=0.93)
plainstates a causen = 164- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Studies where multiple modeling groups attempt to solve the same pre-specified problem have been useful in various fields for code verification and, more generally, for scientific reproducibility [48–50].”
§ Credibility assessment of in silico clinical trials for medical devices > 3. Verification, validation and uncertainty quantification for ISCTs > 3.1. Verification
What the sources say
- Partly supportedcites [48] K Valen-Sendstad, 2018, Real-world variability in the prediction of intracranial aneurysm wall shear stress: the 2015 international aneurysm CFD challenge, Cardiovascular engineering and technology sourceExperimental — accuracy not yet measured
The Methods (Response to the Challenge; page not provided) describe 26 modeling teams submitting CFD results, and the Conclusions report variability that impedes combining results across groups. This supports the study as an example of a multi-group reproducibility challenge, but our copy does not establish that such studies have been useful across various fields or for code verification. (LLM, conf=0.84)
- Supportedcites [49] SA Niederer, 2011, Verification of cardiac tissue electrophysiology simulators using an N-version benchmark, Phil. Trans. R. Soc. A source
Section 2 describes multi-code comparisons of common problems in earth science, astronomy, meteorology and mantle simulation as code-verification strategies. It says the shared-benchmark approach also demonstrates that a benchmark is reproducible; the Benchmark utility section describes this study as confirming the reproducibility of multi-scale simulations. (LLM, conf=0.91)
- Partly supportedcites [50] I Sechopoulos, 2015, Monte Carlo reference data sets for imaging research: Executive summary of the report of AAPM Research Committee Task Group 195, Medical physics sourceExperimental — accuracy not yet measured
The Abstract and Section 2.A describe four Monte Carlo packages applied to the same specified imaging simulations, and the Discussion explains how comparing their results helps benchmark and validate simulation codes. The paper supports that example, but does not establish usefulness across various fields or for scientific reproducibility more generally. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of the ISCT compared favorably with the results of an existing clinical trial that was used as a reference”
§ Credibility assessment of in silico clinical trials for medical devices > 2. In silico clinical trials for medical devices > 2.3. Example ISCTs for medical devices
What the sources say
- Supportedcites [32] A Badano, 2018, Evaluation of digital breast tomosynthesis as replacement of full-field digital mammography using an in silico imaging trial, JAMA network open source
The Results state that VICTRE’s differential performance was consistent with the comparative trial’s aggregate results and its stronger performance for detecting masses than microcalcifications. The Discussion likewise says the overall results were consistent with the comparative trial, supporting the claim that the in silico trial compared favorably with its clinical reference. (LLM, conf=0.93)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“the Virtual Imaging Clinical Trial for Regulatory Evaluation (VICTRE) project [32] explored the possibility of using an ISCT to compare the performance of FFDM and DBT.”
§ Credibility assessment of in silico clinical trials for medical devices > 2. In silico clinical trials for medical devices > 2.3. Example ISCTs for medical devices
What the sources say
- Supportedcites [32] A Badano, 2018, Evaluation of digital breast tomosynthesis as replacement of full-field digital mammography using an in silico imaging trial, JAMA network open source
The Discussion describes VICTRE as an in silico trial comparing digital mammography (DM) with digital breast tomosynthesis (DBT), and reports their differential performance. The paper’s title identifies the mammography modality as full-field digital mammography. Page numbers were not available in our copy. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“First, we noted that with PSMs it is common to perform calculation verification studies such as grid refinement studies using a single, assumed representative, patient.”
§ Credibility assessment of in silico clinical trials for medical devices > 3. Verification, validation and uncertainty quantification for ISCTs > 3.1. Verification
What the sources say
- Supportedcites [9] SK Galappaththige, 2022, Credibility Assessment of Patient-Specific Computational Modeling using Patient-Specific Cardiac Modeling as an Exemplar, PLOS Computational Biology source
Section 2.2 says that in cardiac patient-specific models, often either no verification is performed or a single mesh-resolution study is used to justify the resolution for all patients. This supports the claim that, when such a study is performed, it commonly uses one patient as representative of the others. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
33 more claims · 6 with their cited sources checked
Create a free account to see all 183 claims and to verify the 2 references still open.
- Probably not a substantial claim. 126
- It may state two separate claims as one. 49
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Credibility assessment of in silico clinical trials for medical devices > 2. In silico clinical trials for medical devices > 2.3. Example ISCTs for medical devices
“10K simulated trials of 5K virtual patients each, 50M simulations total”
§ Credibility assessment of in silico clinical trials for medical devices > 2. In silico clinical trials for medical devices > 2.3. Example ISCTs for medical devices
“The study involved 5,000 simulated trials, each comprised of 10,000 virtual patients, for a total of 50 million simulations.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Credibility assessment of in silico clinical trials for medical devices > 3. Verification, validation and uncertainty quantification for ISCTs > 3.2. Validation
“enabling validation of both the virtual patient in isolation and the coupled device-patient model (Fig 2d; see [56] for an example in stent-graft deployment)”
§ Credibility assessment of in silico clinical trials for medical devices > References
“56. O’Hara T, Virag L, Varro A, Rudy Y, Simulation of the undiseased human cardiac ventricular action potential: model formulation and experimental validation.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Credibility assessment of in silico clinical trials for medical devices > 3. Verification, validation and uncertainty quantification for ISCTs > 3.2. Validation
“If subject-specific virtual patients are generated (whether for cohort generation approaches A, B or C in Fig 1), validation may be possible against other data collected from the same subjects–see Fig 2b.”
§ Credibility assessment of in silico clinical trials for medical devices > 3. Verification, validation and uncertainty quantification for ISCTs > 3.2. Validation
“| Validate patient model | Fig 2b | Patient model using nominal or population-averaged parameter values | Population-averaged clinical data | Cardiac modeling [56] | 4 |”
Citations
Cites
An Introduction to Statistical Learning
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) · Springer Texts in Statistics · doi:10.1007/978-1-4614-7138-7 · Cited by 12,106
Physics-informed machine learning
Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., & Yang, L. (2021) · Nature Reviews Physics · doi:10.1038/s42254-021-00314-5 · Cited by 8,718
Understanding bland altman analysis
Giavarina, D. (2015) · Biochemia Medica · doi:10.11613/bm.2015.015 · Cited by 4,706
Verification and Validation in Scientific Computing
Oberkampf, W. L., & Roy, C. J. (2010) · doi:10.1017/cbo9780511760396 · Cited by 1,556
Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models
Zou, K. H., O’Malley, A. J., & Mauri, L. (2007) · Circulation · doi:10.1161/circulationaha.105.594929 · Cited by 1,473
O'Hara, T., Virág, L., Varró, A., & Rudy, Y. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002061 · Cited by 1,196
The ‘Digital Twin’ to enable the vision of precision cardiology
Corral-Acero, J., Margara, F., Marciniak, M., Rodero, C., Loncaric, F., Feng, Y., Gilbert, A., Fernandes, J. F., Bukhari, H. A., Wajdan, A., et al. (2020) · European Heart Journal · doi:10.1093/eurheartj/ehaa159 · Cited by 932
Self-citationExternal validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887
The UVA/PADOVA Type 1 Diabetes Simulator: New Features
Man, C. D., Micheletto, F., Lv, D., Breton, M., Kovatchev, B., & Cobelli, C. (2014) · Journal of Diabetes Science and Technology · doi:10.1177/1932296813514502 · Cited by 791 · cited 1×
“Another example is compartmental glucose-insulin models, which have been used to generate virtual cohorts to evaluate automated insulin pumps [38,39] (more details in Table 1).”
In Silico Preclinical Trials: A Proof of Concept in Closed-Loop Control of Type 1 Diabetes
Kovatchev, B. P., Breton, M., Dalla Man, C., & Cobelli, C. (2009) · Journal of Diabetes Science and Technology · doi:10.1177/193229680900300106 · Cited by 760
Cited by
The future of in silico trials and digital twins in medicine
Samei, E. (2025) · PNAS Nexus · doi:10.1093/pnasnexus/pgaf123 · Cited by 39
Aycock, K. I., Battisti, T., Peterson, A., Jiang, Y. D., Kreuzer, S. M., Capelli, C., Pant, S., Pathmanathan, P., Hoganson, D. M., Levine, S. M., et al. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1433372 · Cited by 40
Self-citationAlharthi, S. (2025) · Saudi Pharmaceutical Journal · doi:10.1007/s44446-025-00055-x · Cited by 43
“…a patient, organ, or biological system that integrates multidimensional, patient-specific data to support clinical decision-making (Drummond & Gonsard 2024), while ISTs refer to structured and comprehensive computational simulations conducted using or in conjunction with DTs (Table 2) (Samei 2025).”
MASLD, MASH, and the CKM Spectrum
Zannad, F. M., Khan, M. S., Bansal, N., Böhm, M., Francque, S., Girerd, N., Januzzi, J. L., Miller, V., Piché, M., Ratziu, V., et al. (2026) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2025.12.015 · Cited by 40
Toward widespread use of virtual trials in medical imaging innovation and regulatory science
Abadi, E., Barufaldi, B., Lago, M. A., Badal, A., Mello-Thoms, C. R., Bottenus, N. B., Wangerin, K. A., Goldburgh, M. M., Tarbox, L., Beaucage‐Gauvreau, E., et al. (2024) · Medical Physics · doi:10.1002/mp.17442 · Cited by 19
Self-citation
Metadata
- Authors
- Pathmanathan, P., Aycock, K., Badal, A., Bighamian, R., Bodner, J., Craven, B. A., & Niederer, S.
- Affiliations
- United States Food and Drug Administration (US)
- Medtronic (United States) (US)
- The Alan Turing Institute (GB)
- Corresponding author
- Pathmanathan, Pras
- ORCID
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2024-08-08
- Volume
- 20
- Issue
- 8
- Pages
- e1012289
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC11309390
- OpenAlex
- W4401418803
- Cited by
- 62
- References
- 70
- Citations by year
- 2026: 30 · 2025: 27 · 2024: 5
- FWCI
- 21.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS Computational Biology (publishedVersion, public-domain)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Simulation-Based Education in Healthcare
- Keywords
- in silico clinical trials, medical devices, credibility assessment, model verification and validation, uncertainty quantification, patient modeling, virtual patients, subject-specific modeling, standardization
- MeSH terms
- Clinical Trials as Topic, Computer Simulation, Equipment and Supplies, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Credibility assessment of in silico clinical trials for medical devices. https://app.cochat.ai/papers/10.1371/journal.pcbi.1012289
- The paper
- Pathmanathan, P., Aycock, K., Badal, A., Bighamian, R., Bodner, J., Craven, B. A., & Niederer, S. (2024). Credibility assessment of in silico clinical trials for medical devices. PLOS Computational Biology. https://doi.org/10.1371/journal.pcbi.1012289
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.