CoChat
Sign Up
Cited by 15Statistical Methods in Clinical TrialsComputational Drug Discovery MethodsPharmaceutical studies and practices

Assessing the performance of QSP models: biology as the driver for validation

Open access
Singh, Fulya Akpinar, Afzal, Nasrin, Smithline, Shepard J., Thalhauser, Craig J.·2024·Journal of Pharmacokinetics and Pharmacodynamics
DOI 10.1007/s10928-023-09871-xPMC PMC11576624
Open this paper in CoChat — free accountOpen PDF

Abstract

Validation of a quantitative model is a critical step in establishing confidence in the model’s suitability for whatever analysis it was designed. While processes for validation are well-established in the statistical sciences, the field of quantitative systems pharmacology (QSP) has taken a more piecemeal approach to defining and demonstrating validation. Although classical statistical methods can be used in a QSP context, proper validation of a mechanistic systems model requires a more nuanced approach to what precisely is being validated, and what role said validation plays in the larger context of the analysis. In this review, we summarize current thoughts of QSP validation in the scientific community, contrast the aims of statistical validation from several contexts (including inference, pharmacometrics analysis, and machine learning) with the challenges faced in QSP analysis, and use examples from published QSP models to define different stages or levels of validation, any of which may be sufficient depending on the context at hand.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10928-023-09871-x

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In regulatory contexts, demonstrating model performance on independent datasets, particularly those relevant to the intended population and CoU, significantly strengthens the case for its reliability.

    “In regulatory contexts, demonstrating model performance on independent datasets, particularly those relevant to the intended population and CoU, significantly strengthens the case for its reliability (11).”

    From Best Practices on QSP Model Reporting for Regulatory Use: perspectives from ISoP QSP SIG Working Group (2026) · cites this paper as [11]

    The sections “Approach to validation in QSP is context-dependent” and “Implications for QSP modeling” say validation should fit the model’s purpose and that testing against data not used in calibration can strengthen confidence in predictions. The paper does not specifically make the claim about regulatory contexts or validation datasets matched to an intended population and context of use; it also cautions that external datasets differ in how much biological validation they provide (pages unavailable in our copy). (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. [1]Allen RJ, Rieger TR, Musante CJ (2016) Efficient generation and selection of virtual populations in quantitative systems pharmacology models. CPT Pharmacomet Syst Pharmacol 5(3):140–146. 10.1002/psp4.12063 PMC4809626 27069777 · DOI
  2. [2]Schmidt BJ, Casey FP, Paterson T, Chan JR (2013) Alternate virtual populations elucidate the type I interferon signature predictive of the response to rituximab in rheumatoid arthritis. BMC Bioinform 14:221. 10.1186/1471-2105-14-221 PMC3717130 23841912 · DOI
  3. [3]Kirouac DC (2018) How do we “validate” a QSP model? CPT Pharmacomet Syst Pharmacol 7(9):547–548. 10.1002/psp4.12310 PMC6157666 29761661 · DOI
  4. [4]Chelliah V, Lazarou G, Bhatnagar S, Gibbs JP, Nijsen M, Ray A, Stoll B, Thompson RA, Gulati A, Soukharev S, Yamada A, Weddell J, Sayama H, Oishi M, Wittemer-Rump S, Patel C, Niederalt C, Burghaus R, Scheerans C, Lippert J, Kabilan S, Kareva I, Belousova N, Rolfe A, Zutshi A, Chenel M, Venezia F, Fouliard S, Oberwittler H, Scholer-Dahirel A, Lelievre H, Bottino D, Collins SC, Nguyen HQ, Wang H, Yoneyama T, Zhu AZX, van der Graaf PH, Kierzek AM (2021) Quantitative systems pharmacology approaches for immuno-oncology: adding virtual patients to the development paradigm. Clin Pharmacol Ther 109(3):605–618. 10.1002/cpt.1987 32686076 PMC7983940 · DOI
  5. [5]Bai JPF, Earp JC, Strauss DG, Zhu H (2020) A perspective on quantitative systems pharmacology applications to clinical drug development. CPT Pharmacomet Syst Pharmacol 9(12):675–677. 10.1002/psp4.12567 PMC7762807 33159491 · DOI
  6. [6]Karelina T, Demin O, Nicholas T, Lu Y, Duvvuri S, Barton HA (2017) A translational systems pharmacology model for abeta kinetics in mouse, monkey, and human. CPT Pharmacomet Syst Pharmacol 6(10):666–675. 10.1002/psp4.12211 PMC5658289 28571112 · DOI
  7. [7]Ramanujan S, Chan JR, Friedrich CM, Thalhauser CJ (2019) A flexible approach for context-dependent assessment of quantitative systems pharmacology models. CPT Pharmacomet Syst Pharmacol 8(6):340–343. 10.1002/psp4.12409 PMC6617835 30983158 · DOI
  8. [8]Braakman S, Pathmanathan P, Moore H (2022) Evaluation framework for systems models. CPT Pharmacomet Syst Pharmacol 11(3):264–289. 10.1002/psp4.12755 PMC8923730 34921743 · DOI
  9. [9]Kirouac DC, Schaefer G, Chan J, Merchant M, Orr C, Huang SA, Moffat J, Liu L, Gadkar K, Ramanujan S (2017) Clinical responses to ERK inhibition in BRAF(V600E)-mutant colorectal cancer predicted using a computational model. NPJ Syst Biol Appl 3:14. 10.1038/s41540-017-0016-1 28649441 PMC5460205 · DOI · Paper page
  10. [10]Klinke DJ 2nd (2008) Integrating epidemiological data into a mechanistic model of type 2 diabetes: validating the prevalence of virtual patients. Ann Biomed Eng 36(2):321–334. 10.1007/s10439-007-9410-y 18046647 · DOI
  11. [11]Milberg O, Gong C, Jafarnejad M, Bartelink IH, Wang B, Vicini P, Narwal R, Roskos L, Popel AS (2019) A QSP model for predicting clinical responses to monotherapy, combination and sequential therapy following CTLA-4, PD-1, and PD-L1 checkpoint blockade. Sci Rep 9(1):11286. 10.1038/s41598-019-47802-4 31375756 PMC6677731 · DOI
  12. [12]Binder M, Sulaimanov N, Clausznitzer D, Schulze M, Huber CM, Lenz SM, Schloder JP, Trippler M, Bartenschlager R, Lohmann V, Kaderali L (2013) Replication vesicles are load- and choke-points in the hepatitis C virus lifecycle. PLoS Pathog 9(8):e1003561. 10.1371/journal.ppat.1003561 23990783 PMC3749965 · DOI
  13. [13]Riggs MM, Bennetts M, van der Graaf PH, Martin SW (2012) Integrated pharmacometrics and systems pharmacology model-based analyses to guide GnRH receptor modulator development for management of endometriosis. CPT Pharmacomet Syst Pharmacol 1:e11. 10.1038/psp.2012.10 PMC3606940 23887363 · DOI
  14. [14]Ouzounoglou E, Kalamatianos D, Emmanouilidou E, Xilouri M, Stefanis L, Vekrellis K, Manolakos ES (2014) In silico modeling of the effects of alpha-synuclein oligomerization on dopaminergic neuronal homeostasis. BMC Syst Biol 8:54. 10.1186/1752-0509-8-54 24885905 PMC4062111 · DOI
  15. [15]Dwivedi G, Fitz L, Hegen M, Martin SW, Harrold J, Heatherington A, Li C (2014) A multiscale model of interleukin-6-mediated immune regulation in Crohn’s disease and its application in drug discovery and development. CPT Pharmacomet Syst Pharmacol 3:e89. 10.1038/psp.2013.64 PMC3910013 24402116 · DOI
  16. [16]Ekerot P, Ferguson D, Glamsta EL, Nilsson LB, Andersson H, Rosqvist S, Visser SA (2013) Systems pharmacology modeling of drug-induced modulation of thyroid hormones in dogs and translation to human. Pharm Res 30(6):1513–1524. 10.1007/s11095-013-0989-4 23568527 · DOI
  17. [17]Kosinsky Y, Dovedi SJ, Peskov K, Voronova V, Chu L, Tomkinson H, Al-Huniti N, Stanski DR, Helmlinger G (2018) Radiation and PD-(L)1 treatment combinations: immune response and dose optimization via a predictive systems model. J Immunother Cancer 6(1):17. 10.1186/s40425-018-0327-9 29486799 PMC5830328 · DOI
  18. [18]Stodtmann S, Nader A, Polepally AR, Suleiman AA, Winzenborg I, Noertersheuser P, Ng J, Mostafa NM, Shebley M (2021) Validation of a quantitative systems pharmacology model of calcium homeostasis using elagolix phase 3 clinical trial data in women with endometriosis. Clin Transl Sci 14(4):1611–1619. 10.1111/cts.13040 33963686 PMC8301592 · DOI
  19. [19]Gadkar K, Budha N, Baruch A, Davis JD, Fielder P, Ramanujan S (2014) A mechanistic systems pharmacology model for prediction of LDL cholesterol lowering by PCSK9 antagonism in human dyslipidemic populations. CPT Pharmacomet Syst Pharmacol 3:e149. 10.1038/psp.2014.47 PMC4260002 25426564 · DOI · Paper page
  20. [20]Wang H, Ma H, Sove RJ, Emens LA, Popel AS (2021) Quantitative systems pharmacology model predictions for efficacy of atezolizumab and nab-paclitaxel in triple-negative breast cancer. J Immunother Cancer. 10.1136/jitc-2020-002100 33579739 PMC7883871 · DOI
  21. [21]Hardiansyah D, Ng CM (2019) Quantitative systems pharmacology model of chimeric antigen receptor T-cell therapy. Clin Transl Sci 12(4):343–349. 10.1111/cts.12636 30990958 PMC6662387 · DOI
  22. [23]Clausznitzer D, Pichardo-Almarza C, Relo AL, van Bergeijk J, van der Kam E, Laplanche L, Benson N, Nijsen M (2018) Quantitative systems pharmacology model for Alzheimer disease indicates targeting sphingolipid dysregulation as potential treatment option. CPT Pharmacomet Syst Pharmacol 7(11):759–770. 10.1002/psp4.12351 PMC6263662 30207429 · DOI
  23. [24]Ma H, Pilvankar M, Wang J, Giragossian C, Popel AS (2021) Quantitative systems pharmacology modeling of PBMC-humanized mouse to facilitate preclinical Immuno-oncology drug development. ACS Pharmacol Transl Sci 4(1):213–225. 10.1021/acsptsci.0c00178 33615174 PMC7887858 · DOI
  24. [25]Hosseini I, Gadkar K, Stefanich E, Li CC, Sun LL, Chu YW, Ramanujan S (2020) Mitigating the risk of cytokine release syndrome in a phase I trial of CD20/CD3 bispecific antibody mosunetuzumab in NHL: impact of translational system modeling. NPJ Syst Biol Appl 6(1):28. 10.1038/s41540-020-00145-7 32859946 PMC7455723 · DOI
  25. [26]Betts A, Haddish-Berhane N, Shah DK, van der Graaf PH, Barletta F, King L, Clark T, Kamperschroer C, Root A, Hooper A, Chen X (2019) A translational quantitative systems pharmacology model for CD3 bispecific molecules: application to quantify T cell-mediated tumor cell killing by P-Cadherin LP DART((R)). AAPS J 21(4):66. 10.1208/s12248-019-0332-z 31119428 PMC6531394 · DOI
  26. [27]Chen X, Hickling TP, Vicini P (2014) A mechanistic, multiscale mathematical model of immunogenicity for therapeutic proteins: part 2-model applications. CPT Pharmacomet Syst Pharmacol 3:e134. 10.1038/psp.2014.31 PMC4211264 25184734 · DOI · Paper page
  27. [28]Biliouris K, Lavielle M, Trame MN (2015) MatVPC: a user-friendly MATLAB-based tool for the simulation and evaluation of systems pharmacology models. CPT Pharmacomet Syst Pharmacol 4(9):547–557. 10.1002/psp4.12011 PMC4592534 26451334 · DOI
  28. [29]Hasegawa C, Duffull SB (2017) Selection and qualification of simplified QSP models when using model order reduction techniques. AAPS J 20(1):2. 10.1208/s12248-017-0170-9 29181592 · DOI
  29. [30]Bansal L, Nichols EM, Howsmon DP, Neisen J, Bessant CM, Cunningham F, Petit-Frere S, Ludbrook S, Damian V (2022) Mathematical modeling of complement pathway dynamics for target validation and selection of drug modalities for complement therapies. Front Pharmacol 13:855743. 10.3389/fphar.2022.855743 35517827 PMC9061988 · DOI
  30. [31]Helmlinger G, Sokolov V, Peskov K, Hallow KM, Kosinsky Y, Voronova V, Chu L, Yakovleva T, Azarov I, Kaschek D, Dolgun A, Schmidt H, Boulton DW, Penland RC (2019) Quantitative systems pharmacology: an exemplar model-building workflow with applications in cardiovascular, metabolic, and oncology drug development. CPT Pharmacomet Syst Pharmacol 8(6):380–395. 10.1002/psp4.12426 PMC6617832 31087533 · DOI
  31. [32]Cheng Y, Thalhauser CJ, Smithline S, Pagidala J, Miladinov M, Vezina HE, Gupta M, Leil TA, Schmidt BJ (2017) QSP Toolbox: computational implementation of Integrated Workflow Components for deploying Multi-Scale mechanistic models. AAPS J 19(4):1002–1016. 10.1208/s12248-017-0100-x 28540623 · DOI · Paper page
  32. [33]Gareth James DWTHRT (2013) An introduction to statistical learning: with applications in R. Springer, New York
  33. [34]Colby E, Bair E (2013) Cross-validation for nonlinear mixed effects models. J Pharmacokinet Pharmacodyn 40(2):243–252. 10.1007/s10928-013-9313-5 23532511 PMC3668859 · DOI
  34. [36]Dhaese SAM, Farkas A, Colin P, Lipman J, Stove V, Verstraete AG, Roberts JA, De Waele JJ (2019) Population pharmacokinetics and evaluation of the predictive performance of pharmacokinetic models in critically ill patients receiving continuous infusion meropenem: a comparison of eight pharmacokinetic models. J Antimicrob Chemother 74(2):432–441. 10.1093/jac/dky434 30376103 · DOI
  35. [37]Ryu S, Jung WJ, Jiao Z, Chae J-W, Yun H-y (2020) External validation of the predictive performance of population pharmacokinetic models for phenobarbital in pediatric patients. medRxiv. 10.1101/2020.09.10.20192005 · DOI
  36. [38]Goodfellow Ia (2016) Deep learning. The MIT Press, Cambridge
  37. [39]Gessert N, Nielsen M, Shaikh M, Werner R, Schlaefer A (2020) Skin lesion classification using ensembles of multi-resolution EfficientNets with meta data. MethodsX 7:100864. 10.1016/j.mex.2020.100864 32292713 PMC7150512 · DOI
  38. [40]Zhang C, Bengio S, Hardt M, Recht B, Vinyals O (2021) Understanding deep learning (still) requires rethinking generalization. Commun ACM 64(3):107–115. 10.1145/3446776 · DOI · Paper page
  39. [41]Belkin M, Hsu D, Ma S, Mandal S (2019) Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proc Natl Acad Sci USA 116(32):15849–15854. 10.1073/pnas.1903070116 31341078 PMC6689936 · DOI · Paper page
  40. [42]Roesch E, Rackauckas C, Stumpf MPH (2021) Collocation based training of neural ordinary differential equations. Stat Appl Genet Mol Biol 20(2):37–49. 10.1515/sagmb-2020-0025 34237805 · DOI
  41. [44]Hamuro L, Tirucherai GS, Crawford SM, Nayeem A, Pillutla RC, DeSilva BS, Leil TA, Thalhauser CJ (2019) Evaluating a Multiscale mechanistic model of the Immune System to predict human immunogenicity for a Biotherapeutic in Phase 1. AAPS J 21(5):94. 10.1208/s12248-019-0361-7 31342199 · DOI
  42. [45]Chen X, Hickling TP, Vicini P (2014) A mechanistic, multiscale mathematical model of immunogenicity for therapeutic proteins: part 1-theoretical model. CPT Pharmacomet Syst Pharmacol 3:e133. 10.1038/psp.2014.30 PMC4211265 25184733 · DOI

Cited by

Sorted by citations

Authors
Singh, F. A., Afzal, N., Smithline, S. J., & Thalhauser, C. J.
Affiliations
Genmab (United States) (US)
Corresponding author
Thalhauser, Craig J.
ORCID
Venue
Journal of Pharmacokinetics and Pharmacodynamics
Publisher
Springer Science and Business Media LLC
Published
2023-06-29
Volume
51
Issue
5
Pages
533-542
ISSN
  • 1567-567X (ISSN-L)
  • 1573-8744
DOI
10.1007/s10928-023-09871-x
PMC
PMC11576624
OpenAlex
W4382600581
Cited by
15
References
3
Citations by year
2026: 5 · 2025: 8 · 2024: 2
FWCI
0.62
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Pharmacokinetics and Pharmacodynamics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
  • Health Sciences › Medicine › Pediatrics, Perinatology and Child Health › Pharmaceutical studies and practices
Keywords
quantitative systems pharmacology, model validation, statistical validation, pharmacometrics, machine learning validation, model evaluation
MeSH terms
Machine Learning, Animals, Humans, Models, Biological, Reproducibility of Results, Systems Biology, Validation Studies as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review