The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
In silico trials: Verification, validation and uncertainty quantification of predictive models used in the regulatory evaluation of biomedical products
Open accessAbstract
Historically, the evidences of safety and efficacy that companies provide to regulatory agencies as support to the request for marketing authorization of a new medical product have been produced experimentally, either in vitro or in vivo. More recently, regulatory agencies started receiving and accepting evidences obtained in silico, i.e. through modelling and simulation. However, before any method (experimental or computational) can be acceptable for regulatory submission, the method itself must be considered "qualified" by the regulatory agency. This involves the assessment of the overall "credibility" that such a method has in providing specific evidence for a given regulatory procedure. In this paper, we describe a methodological framework for the credibility assessment of computational models built using mechanistic knowledge of physical and chemical phenomena, in addition to available biological and physiological knowledge; these are sometimes referred to as "biophysical" models. Using guiding examples, we explore the definition of the context of use, the risk analysis for the definition of the acceptability thresholds, and the various steps of a comprehensive verification, validation and uncertainty quantification process, to conclude with considerations on the credibility of a prediction for a specific context of use. While this paper does not provide a guideline for the formal qualification process, which only the regulatory agencies can provide, we expect it to help researchers to better appreciate the extent of scrutiny required, which should be considered early on in the development/use of any (new) in silico evidence.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Of particular relevance for the field of systems biology is The European Medicines Agency’s (EMA) Guideline on the Reporting of Physiologically Based Pharmacokinetic (PBPK) Modeling and Simulation issued in 2018 [9, 65].”
From Adapting modeling and simulation credibility standards to computational systems biology (2023) · cites this paper as [65]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Verification and Validation in Computational Science and Engineering
Roache, P. J. (1998) · Cited by 2,160
Verification and Validation in Scientific Computing
Oberkampf, W. L., & Roy, C. J. (2010) · doi:10.1017/cbo9780511760396 · Cited by 1,556 · cited 3×
“The reader is referred to ASME VV-40-2018 for a complete description of all credibility factors and to ASME VV-10-2007,4 ASME VV-202009,5 Oberkampf and Roy [19], and Roache [20] for more detailed information on model verification, validation, and uncertainty quantification.”
BackgroundMethodNonlinear System Identification: NARMAX Methods in the Time, Frequency, and Spatio-Temporal Domains
Billings, S. A. (2013) · CERN Document Server (European Organization for Nuclear Research) · Cited by 655
Sager, P. T., Gintant, G. A., Turner, J. R., Pettit, S. D., & Stockbridge, N. (2013) · American Heart Journal · doi:10.1016/j.ahj.2013.11.004 · Cited by 541 · cited 1×
“CiPA has become a global effort, involving many industry and academic partners, in addition to regulators [18,3].”
BackgroundComputational models in cardiology
Niederer, S., Lumens, J., & Trayanova, N. A. (2018) · Nature Reviews Cardiology · doi:10.1038/s41569-018-0104-y · Cited by 435 · cited 1×
“Current human cardiac electrophysiology models integrate detailed information on the dynamic processes underlying cardiac electrical excitation from subcellular to whole organ levels [14].”
BackgroundExtracting clinically relevant data from finite element simulations
Viceconti, M., Olsen, S., Nolte, L.-P., & Burton, K. A. (2005) · Clinical Biomechanics · doi:10.1016/j.clinbiomech.2005.01.010 · Cited by 343 · cited 1×
“But when the predictions of a computer model are used to make clinical recommendations, there is implicit risk to the patient(s) associated with that computer model; and therefore it is necessary to demonstrate the credibility of the model is commensurate with the patient risk [48].”
MethodSelf-citationLearning under Concept Drift: an OverviewPreprint
Žliobaitė, I. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1010.4784 · Cited by 332 · cited 1×
“Indeed, a recent FDA proposal outlining a regulatory framework for Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) [29] suggested that these models should be continuously tested in order to avoid issues such as context drift [30].”
BackgroundPassini, E., Britton, O. J., Lu, H. R., Rohrbacher, J., Hermans, A. N., Gallacher, D. J., Greig, R. J. H., Bueno‐Orovio, A., & Rodríguez, B. (2017) · Frontiers in Physiology · doi:10.3389/fphys.2017.00668 · Cited by 326 · cited 1×
“This class of trial exploits human-based modelling and simulation technologies for virtual testing of pharmacological therapies [7] and devices [8].”
BackgroundIn silico clinical trials: how computer simulation will transform the biomedical industry
Viceconti, M., HENNEY, A. M., & Morley‐Fletcher, E. (2016) · International Journal of Clinical Trials · doi:10.18203/2349-3259.ijct20161408 · Cited by 316
Self-citationIn silico clinical trials: concepts and early adoptions
Pappalardo, F., Russo, G. L., Musuamba, F. T., & Viceconti, M. (2018) · Briefings in Bioinformatics · doi:10.1093/bib/bby043 · Cited by 281 · cited 2×
“Thus, human in silico clinical trials are now emerging as an important paradigm in the development of medical therapies [6].”
BackgroundSelf-citationComputational biology in the study of cardiac ion channels and cell electrophysiology
Rudy, Y., & Silva, J. R. (2006) · Quarterly Reviews of Biophysics · doi:10.1017/s0033583506004227 · Cited by 262 · cited 1×
“Utilizing these frameworks, modelling and simulation studies have played a central role in the discovery of cardiac arrhythmia mechanisms [15,16] and treatments such as electrical defibrillation [17].”
MethodApplying machine learning techniques for ADME-Tox prediction: a review
Maltarollo, V. G., Gertrudes, J. C., Oliveira, P. R., & Honorio, K. M. (2014) · Expert Opinion on Drug Metabolism & Toxicology · doi:10.1517/17425255.2015.980814 · Cited by 176 · cited 1×
“Predictive models developed using traditional frequentist statistics [23 – 25], as well as machine learning models [26 – 28], do not rely in general on prior knowledge; thus, they are not a ff ected by epistemic error.”
MethodGong, C., Milberg, O., Wang, B., Vicini, P., Narwal, R., Roskos, L. K., & Popel, A. S. (2017) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0320 · Cited by 169
Verification, validation and accreditation of simulation models
Sargent, R. G. (2002) · Winter Simulation Conference · doi:10.1109/wsc.2000.899697 · Cited by 135 · cited 1×
“The credibility assessment of ABM is a complex topic; some extensive discussion can be found here [42].”
BackgroundIn silico cancer research towards 3R
Jean-Quartier, C., Jeanquartier, F., Jurišica, I., & Holzinger, A. (2018) · BMC Cancer · doi:10.1186/s12885-018-4302-0 · Cited by 132 · cited 1×
“In addition, modelling and simulation is being used to reduce, refine and replace animal experimentation [9,10], and even to replace bench tests [11,12].”
MethodMottini, C., Napolitano, F., Li, Z., Gao, X., & Cardone, L. (2019) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2019.09.023 · Cited by 125 · cited 1×
“Predictive models developed using traditional frequentist statistics [23 – 25], as well as machine learning models [26 – 28], do not rely in general on prior knowledge; thus, they are not a ff ected by epistemic error.”
MethodUnderstanding the time course of pharmacological effect: a PKPD approach
Wright, D. F. B., Winter, H. R., & Duffull, S. B. (2011) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.2011.03925.x · Cited by 99 · cited 1×
“Predictive models developed using traditional frequentist statistics [23 – 25], as well as machine learning models [26 – 28], do not rely in general on prior knowledge; thus, they are not a ff ected by epistemic error.”
MethodBender, B. C., Schindler, E., & Friberg, L. E. (2013) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.12258 · Cited by 99 · cited 1×
“8 Historically, quali fi cation focused on experimental or (pharmaco) statistical methods, such as population pharmacokinetics (popPK), pharmacokinetic-pharmacodynamics (PK/PD), and dose-exposure response (DER) models [44].”
BackgroundApplicability Analysis of Validation Evidence for Biomedical Computational Models
Pathmanathan, P., Gray, R. A., Romero, V. J., & Morrison, T. M. (2017) · Journal of Verification Validation and Uncertainty Quantification · doi:10.1115/1.4037671 · Cited by 68 · cited 1×
“proposed a step-bystep applicability analysis that can facilitate the identification of gaps between validation evidence and a COU [22].”
Morrison, T. M., Hariharan, P., Funkhouser, C. M., Afshari, P., Goodin, M., & Horner, M. (2019) · ASAIO Journal · doi:10.1097/mat.0000000000000996 · Cited by 68 · cited 3×
“In addition to addressing potential gaps between input system parameters, assessing the applicability also requires careful consideration of the gap between the measured validation outputs and each QOI [21].”
BackgroundInfluentialCredibility of In Silico Trial Technologies—A Theoretical Framing
Viceconti, M., Juarez, M. A., Curreli, C., Pennisi, M., Russo, G., & Pappalardo, F. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2949888 · Cited by 67 · cited 1×
“this into a proper theoretical framework [45].”
Self-citationComputational Modeling of PI3K/AKT and MAPK Signaling Pathways in Melanoma Cancer
Pappalardo, F., Russo, G. L., Candido, S., Pennisi, M. A., Cavalieri, S., Motta, S., McCubrey, J. A., Nicoletti, F., & Libra, M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0152104 · Cited by 66
Self-citationHaddad, T., Himes, A., Thompson, L., Irony, T., Nair, R., & on Behalf of MDIC Computer Modeling and Simulation Working Group Participants (2017) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543406.2017.1300907 · Cited by 62 · cited 1×
“An interesting use of digital patient cohorts is to provide a prior in Bayesian adaptive clinical trial designs [47].”
MethodNorton, K., Wallace, T., Pandey, N. B., & Popel, A. S. (2017) · BMC Systems Biology · doi:10.1186/s12918-017-0445-x · Cited by 52
Rodríguez, B., Carusi, A., Abi‐Gerges, N., Ariga, R., Britton, O. J., Bub, G., Bueno‐Orovio, A., Burton, R. A. B., Carapella, V., Cardone-Noott, L., et al. (2015) · EP Europace · doi:10.1093/europace/euv320 · Cited by 49
A computational model to predict the immune system activation by citrus-derived vaccine adjuvants
Pappalardo, F., Fichera, E., Paparone, N., Lombardo, A., Pennisi, M. A., Russo, G. L., Leotta, M., Pappalardo, F., Pedretti, A., Fiore, F. D., et al. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw293 · Cited by 47
Self-citationIntravascular Stents in the Last and the Next 10 Years
Palmaz, J. C. (2004) · Journal of Endovascular Therapy · doi:10.1177/15266028040110s621 · Cited by 47 · cited 1×
“However, the most common instance is a medical device that releases some pharmaceutical substance as part of its function; a good example are drug-eluting stents [43].”
BackgroundIntravascular Stents in the Last and the Next 10 Years
Palmaz, J. C. (2004) · Journal of Endovascular Therapy · doi:10.1583/04-1363.1 · Cited by 45
Pellizzer, E. P., Lemos, C. A. A., Almeida, D. A. D. F., Batista, V. E. D. S., Santiago, J. F., & Verri, F. R. (2018) · Materials Science and Engineering C · doi:10.1016/j.msec.2018.05.012 · Cited by 44 · cited 1×
“In addition, modelling and simulation is being used to reduce, refine and replace animal experimentation [9,10], and even to replace bench tests [11,12].”
MethodParvinian, B., Pathmanathan, P., Daluwatte, C., Yaghouby, F., Gray, R. A., Weininger, S., Morrison, T. M., & Scully, C. G. (2019) · Frontiers in Physiology · doi:10.3389/fphys.2019.00220 · Cited by 44
Radio-Frequency Safety Assessment of Stents in Blood Vessels During Magnetic Resonance Imaging
Fujimoto, K., Angelone, L. M., Lucano, E., Rajan, S. S., & Iacono, M. I. (2018) · Frontiers in Physiology · doi:10.3389/fphys.2018.01439 · Cited by 37 · cited 1×
“A large body of research across biomedicine has expanded the credibility of modelling and simulation beyond academia, with dynamic activity also in regulatory agencies and industry [3 – 5].”
BackgroundTrayanova, N. A., Plank, G., & Rodríguez, B. (2006) · Heart Rhythm · doi:10.1016/j.hrthm.2006.04.015 · Cited by 34 · cited 1×
“Utilizing these frameworks, modelling and simulation studies have played a central role in the discovery of cardiac arrhythmia mechanisms [15,16] and treatments such as electrical de fi - brillation [17].”
MethodSuccesses and failures in modeling heart cell electrophysiology
Noble, D. (2011) · Heart Rhythm · doi:10.1016/j.hrthm.2011.06.014 · Cited by 33 · cited 1×
“Utilizing these frameworks, modelling and simulation studies have played a central role in the discovery of cardiac arrhythmia mechanisms [15,16] and treatments such as electrical defibrillation [17].”
MethodFrom bed to bench: How in silico medicine can help ageing research
Viceconti, M., & Dall’Ara, E. (2018) · Mechanisms of Ageing and Development · doi:10.1016/j.mad.2018.07.001 · Cited by 33
Self-citationPennisi, M. A., Russo, G. L., Motta, S., & Pappalardo, F. (2015) · Journal of Immunological Methods · doi:10.1016/j.jim.2015.08.014 · Cited by 28
Self-citationAllegaert, K., Flint, R. B., & Smits, A. (2019) · Expert Opinion on Drug Metabolism & Toxicology · doi:10.1080/17425255.2019.1655540 · Cited by 27 · cited 1×
“In Bayesian models [31 – 33], the posterior probability is the sum of the prior probability (which if informed by a priori knowledge can be assumed to a ff ect primarily the epistemic error) and the likelihood (which is informed by observational data, and thus is a ff ected by aleatoric errors).”
BackgroundAl-Sallami, H. S., Newall, F. H., Monagle, P., Ignjatović, V., Cranswick, N., & Duffull, S. B. (2016) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.12930 · Cited by 22
Watson, W. A., Rhodes, N. J., Echenique, I. A., Angarone, M., & Scheetz, M. H. (2017) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1111/jcpt.12520 · Cited by 22
Local Endovascular Delivery, Gene Therapy, and Cell Transplantation for Peripheral Arterial Disease
Opie, S. R., & Dib, N. (2004) · Journal of Endovascular Therapy · doi:10.1177/15266028040110s617 · Cited by 19
A Data-Driven Framework for Identifying Nonlinear Dynamic Models of Genetic Parts
Krishnanathan, K., Anderson, S., Billings, S. A., & Kadirkamanathan, V. (2012) · ACS Synthetic Biology · doi:10.1021/sb300009t · Cited by 19 · cited 1×
“A good example is the Nonlinear AutoRegressive Moving Average with eXogenous input (NARMAX) model [34,35].”
MethodComputational modeling in melanoma for novel drug discovery
Pennisi, M. A., Russo, G. L., Salvatore, V. D., Candido, S., Libra, M., & Pappalardo, F. (2016) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2016.1174688 · Cited by 18
Self-citationThe Application of Machine Learning Techniques in Clinical Drug Therapy
Meng, H., Jin, W., Yan, C., & Yang, H. (2018) · Current Computer - Aided Drug Design · doi:10.2174/1573409914666180525124608 · Cited by 16
in silico Clinical Trials: How Computer Simulation will Transform the Biomedical Industry
Viceconti, M., HENNEY, A. M., & Morley‐Fletcher, E. (2016) · doi:10.13140/rg.2.1.2756.6164 · Cited by 16
Self-citationHunter, P. J., & Smith, N. P. (2016) · The Journal of Physiology · doi:10.1113/jp273415 · Cited by 15 · cited 1×
“Computer modelling and simulation of humans in both health and disease is a powerful tool in biomedical research, augmenting experimental and clinical research through detailed mechanistic and systematic investigations which are impossible with other means [1,2].”
BackgroundAntihistaminic effects of rupatadine and PKPD modelling
Peña, J., Carbó, M., Solans, A., Nadal, T., Izquierdo, I., & Merlos, M. (2008) · European Journal of Drug Metabolism and Pharmacokinetics · doi:10.1007/bf03191027 · Cited by 12 · cited 1×
“Predictive models developed using traditional frequentist statistics [23 – 25], as well as machine learning models [26 – 28], do not rely in general on prior knowledge; thus, they are not a ff ected by epistemic error.”
MethodSearch for an Optimal Design of a Bioprosthetic Venous Valve: In silico and in vitro Studies
Chen, H. Y., Tien, W.-S., Chambers, S. D., Dabiri, D., & Kassab, G. S. (2019) · European Journal of Vascular and Endovascular Surgery · doi:10.1016/j.ejvs.2018.12.008 · Cited by 12 · cited 1×
“In addition, modelling and simulation is being used to reduce, refine and replace animal experimentation [9,10], and even to replace bench tests [11,12].”
MethodA Bayesian model for evaluating specificity of treatment effects in clinical trials
Simon, R. M., Dixon, D. O., & Freidlin, B. (1995) · Cancer treatment and research · doi:10.1007/978-1-4615-2009-2_8 · Cited by 11
Editorial: Safety Pharmacology – Risk Assessment QT Interval Prolongation and Beyond
Grandi, E., Morotti, S., Pueyo, E., & Rodríguez, B. (2018) · Frontiers in Physiology · doi:10.3389/fphys.2018.00678 · Cited by 10 · cited 1×
“CiPA has become a global effort, involving many industry and academic partners, in addition to regulators [18,3].”
BackgroundBrown, J. E., Qiang, R., Stadnik, P. J., Stotts, L. J., & Arx, J. A. V. (2017) · IEEE Pulse · doi:10.1109/mpul.2017.2701258 · Cited by 3 · cited 1×
“This class of trial exploits human-based modelling and simulation technologies for virtual testing of pharmacological therapies [7] and devices [8].”
Background
Research works printed but not linked: 1
- A.R. MacLeod, H.S. Gill, Generating preclinical evidence for MHRA - an in silico clinical trial examining the safety of a novel device for knee arthritis treatment,” presented at the Bath Biomechanics Symposium 2019, Bath, 2019.
Cited by
Sorted by citations
Sabe, V. T., Ntombela, T., Jhamba, L. A., Maguire, G. E. M., Govender, T., Naicker, T., & Kruger, H. G. (2021) · European Journal of Medicinal Chemistry · doi:10.1016/j.ejmech.2021.113705 · Cited by 665
Musuamba, F. T., Rusten, I. S., Lesage, R., Russo, G. L., Bursi, R., Emili, L., Wangorsch, G., Manolis, E., Karlsson, K. E., Kulesza, A. J., et al. (2021) · CPT Pharmacometrics & Systems Pharmacology · doi:10.1002/psp4.12669 · Cited by 175
Self-citationArtificial intelligence and machine learning applications in biopharmaceutical manufacturing
Rathore, A. S., Nikita, S., Thakur, G., & Mishra, S. (2022) · Trends in biotechnology · doi:10.1016/j.tibtech.2022.08.007 · Cited by 174
Finite Element Analysis of Fracture Fixation
Lewis, G. S., Mischler, D., Wee, H., Reid, J. S., & Varga, P. (2021) · Current Osteoporosis Reports · doi:10.1007/s11914-021-00690-y · Cited by 152
Emerging technologies and their impact on regulatory science
Anklam, E., Bahl, M. I., Ball, R., Beger, R. D., Cohen, J. E., Fitzpatrick, S., Girard, P., Halamoda‐Kenzaoui, B., Hinton, D., Hirose, A., et al. (2021) · Experimental Biology and Medicine · doi:10.1177/15353702211052280 · Cited by 134
Metadata
- Authors
- Viceconti, M., Pappalardo, F., Rodriguez, B., Horner, M., Bischoff, J., & Musuamba Tshinanu, F.
- Affiliations
- Istituto Ortopedico Rizzoli (IT)
- University of Catania (IT)
- Corresponding author
- Viceconti, Marco
- Venue
- Methods
- Publisher
- Elsevier BV
- Published
- 2021-01
- Volume
- 185
- Pages
- 120-127
- ISSN
- 1046-2023 (ISSN-L)
- PMC
- PMC7883933
- OpenAlex
- W3001465925
- Cited by
- 361
- References
- 48
- Citations by year
- 2026: 85 · 2025: 71 · 2024: 72 · 2023: 48 · 2022: 33 · 2021: 44
- FWCI
- 31.72
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Methods (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, cc-by-nc-nd)
- INFM-OAR (INFN Catania) (submittedVersion, cc-by)
- Funders
- Wellcome Trust
- Topics
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- verification and validation, uncertainty quantification, biophysical model, regulatory evaluation, drug approval, context of use, risk analysis, acceptability threshold, prediction model, regulatory qualification
- MeSH terms
- Machine Learning, Computer Simulation, Models, Theoretical, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
