CoChat
Sign Up
Cited by 67Health Systems, Economic Evaluations, Quality of LifeMathematical and Theoretical Epidemiology and Ecology ModelsClinical practice guidelines implementation

Guidelines for multi-model comparisons of the impact of infectious disease interventions

Open access
den Boon, Saskia, Jit, Mark, Brisson, Marc, Medley, Graham, Beutels, Philippe, White, Richard, Flasche, Stefan, Hollingsworth, T. Déirdre +7 more·2019·BMC Medicine
DOI 10.1186/s12916-019-1403-9PMC PMC6699075
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Despite the increasing popularity of multi-model comparison studies and their ability to inform policy recommendations, clear guidance on how to conduct multi-model comparisons is not available. Herein, we present guidelines to provide a structured approach to comparisons of multiple models of interventions against infectious diseases. The primary target audience for these guidelines are researchers carrying out model comparison studies and policy-makers using model comparison studies to inform policy decisions. METHODS: The consensus process used for the development of the guidelines included a systematic review of existing model comparison studies on effectiveness and cost-effectiveness of vaccination, a 2-day meeting and guideline development workshop during which mathematical modellers from different disease areas critically discussed and debated the guideline content and wording, and several rounds of comments on sequential versions of the guidelines by all authors. RESULTS: The guidelines provide principles for multi-model comparisons, with specific practice statements on what modellers should do for six domains. The guidelines provide explanation and elaboration of the principles and practice statements as well as some examples to illustrate these. The principles are (1) the policy and research question - the model comparison should address a relevant, clearly defined policy question; (2) model identification and selection - the identification and selection of models for inclusion in the model comparison should be transparent and minimise selection bias; (3) harmonisation - standardisation of input data and outputs should be determined by the research question and value of the effort needed for this step; (4) exploring variability - between- and within-model variability and uncertainty should be explored; (5) presenting and pooling results - results should be presented in an appropriate way to support decision-making; and (6) interpretation - results should be interpreted to inform the policy question. CONCLUSION: These guidelines should help researchers plan, conduct and report model comparisons of infectious diseases and related interventions in a systematic and structured manner for the purpose of supporting health policy decisions. Adherence to these guidelines will contribute to greater consistency and objectivity in the approach and methods used in multi-model comparisons, and as such improve the quality of modelled evidence for policy.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12916-019-1403-9

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. [1]Egger M Johnson L Althaus C Schöni A Salanti G Low N Norris S Developing WHO guidelines: time to formally include evidence from mathematical modelling studies F1000Res 2018 6 1584 10.12688/f1000research.12367.2 PMC5829466 29552335 · DOI
  2. [2]Hollingsworth TD Medley GF Learning from multi-model comparisons: collaboration leads to insights, but limitations remain Epidemics. 2017 18 1 3 10.1016/j.epidem.2017.02.014 28279450 · DOI
  3. [3]Jit M Brisson M Modelling the epidemiology of infectious diseases for decision analysis: a primer Pharmacoeconomics. 2011 29 371 386 10.2165/11539960-000000000-00000 21504239 PMC7100690 · DOI
  4. [4]Vynnycky E Fine PE The natural history of tuberculosis: the implications of age-dependent risks of disease and the role of reinfection Epidemiol Infect 1997 119 2 183 201 10.1017/S0950268897007917 9363017 PMC2808840 · DOI
  5. [5]Kim JJ Brisson M Edmunds WJ Goldie SJ Modeling cervical cancer prevention in developed countries Vaccine. 2008 26 Suppl 10 K76 K86 10.1016/j.vaccine.2008.06.009 18847560 PMC2769256 · DOI
  6. [6]Drolet M Bénard É Jit M Hutubessy R Brisson M Model comparisons of the effectiveness and cost-effectiveness of vaccination: a systematic review of the literature Value Health 2018 21 10 1250 1258 10.1016/j.jval.2018.03.014 30314627 · DOI
  7. [7]Mount Hood 4 Modeling Group Computer modelling of diabetes and its complications. A report on the fourth Mount Hood challenge meeting Diabetes Care 2007 30 6 1638 1646 10.2337/dc07-9919 17526823 · DOI
  8. [8]Berry DA Cronin KA Plevritis SK Fryback DG Clarke L Zelen M Effect of screening and adjuvant therapy on mortality from breast cancer N Engl J Med 2005 353 1784 1792 10.1056/NEJMoa050518 16251534 · DOI
  9. [9]Hoogendoorn M Feenstra TL Asukai Y Borg S Hansen RN Jansson S-A Cost-effectiveness models for chronic obstructive pulmonary disease: cross-model comparison of hypothetical treatment scenarios Value Health 2014 17 525 536 10.1016/j.jval.2014.03.1721 25128045 · DOI
  10. [10]UNAIDS/WHO/SACEMA Expert Group on Modelling the Impact and Cost of Male Circumcision for HIV Prevention Male circumcision for HIV prevention in high HIV prevalence settings: what can mathematical modelling contribute to informed decision making? PLoS Med 2009 6 9 e10000109 10.1371/journal.pmed.1000109 PMC2731851 19901974 · DOI
  11. [11]Eaton JW Menzies NA Stover J Cambiano V Chindelevitch L Cori A Health benefits, costs, and cost-effectiveness of earlier eligibility for adult antiretroviral therapy and expanded treatment coverage: a combined analysis of 12 mathematical models Lancet Glob Health 2013 2 1 23 34 10.1016/S2214-109X(13)70172-4 25083415 PMC4114402 · DOI
  12. [12]Houben R Menzies NA Sumner T Huynh GH Arinaminpathy N Goldhaber-Fiebert JD Feasibility of achieving the 2025 WHO global tuberculosis targets in South Africa, China, and India: a combined analysis of 11 mathematical models Lancet Global Health 2016 4 11 E806 E815 10.1016/S2214-109X(16)30199-1 27720688 PMC6375908 · DOI
  13. [13]Smith ME Singh BK Irvine MA Stolk WA Subramanian S Hollingsworth TD Michael E Predicting lymphatic filariasis transmission and elimination dynamics using a multi-model ensemble framework Epidemics. 2017 18 16 28 10.1016/j.epidem.2017.02.006 28279452 PMC5340857 · DOI
  14. [14]Halloran ME Ferguson NM Eubank S Longini IM Jr Cummings DAT Lewis B Modeling targeted layered containment of an influenza pandemic in the United States Proc Natl Acad Sci U S A 2008 105 4639 4644 10.1073/pnas.0706849105 18332436 PMC2290797 · DOI
  15. [15]Althaus CL Turner KME Schmid BV Heijne JCM Kretzschmar M Low N Transmission of Chlamydia trachomatis through sexual partnerships: a comparison between three individual-based models and empirical data J R Soc Interface 2012 9 36 146 10.1098/rsif.2011.0131 PMC3223622 21653569 · DOI
  16. [17]Hutubessy R Henao AM Namgyal P Moorthy V Hombach J Results from evaluations of models and cost-effectiveness tools to support introduction decisions for new vaccines need critical appraisal BMC Med 2011 9 55 10.1186/1741-7015-9-55 21569407 PMC3117725 · DOI
  17. [18]Chaiyakunapruk N Somkrua R Hutubessy R Henao AM Hombach J Melegaro A Cost effectiveness of pediatric pneumococcal conjugate vaccines: a comparative assessment of decision-making tools BMC Med 2011 9 53 10.1186/1741-7015-9-53 21569402 PMC3117724 · DOI
  18. [19]Postma MJ Jit M Rozenbaum MH Standaert B Tu HA Hutubessy RC Comparative review of three cost-effectiveness models for rotavirus vaccines in national immunization programs; a generic approach applied to various regions in the world BMC Med 2011 9 84 10.1186/1741-7015-9-84 21740545 PMC3158747 · DOI
  19. [20]Jit M Demarteau N Elbasha E Ginsberg G Kim J Praditsitthikorn N Human papillomavirus vaccine introduction in low-income and middle-income countries: guidance on the use of cost-effectiveness models BMC Med 2011 9 54 10.1186/1741-7015-9-54 21569406 PMC3123559 · DOI
  20. [21]Penny MA Verity R Bever CA Sauboin C Galactionova K Flasche S Public health impact and cost-effectiveness of the RTS,S/AS01 malaria vaccine: a systematic comparison of predictions from four mathematical models Lancet. 2016 387 10016 367 375 10.1016/S0140-6736(15)00725-4 26549466 PMC4723722 · DOI
  21. [22]Flasche S Jit M Rodriquez-Barraquer I Coudeville L Recker M Koelle K The long-term safety, public health impact, and cost-effectiveness of routine vaccination with a recombinant, live-attenuated dengue vaccine (Dengvaxia): a model comparison study PLoS Med 2016 13 11 e1002181 10.1371/journal.pmed.1002181 27898668 PMC5127514 · DOI
  22. [23]Malaria vaccine WHO position paper – January 2016 Wkly Epidemiol Rec 2016 91 4 33 52 26829826
  23. [24]Human papillomavirus vaccines WHO position paper, May 2017 Wkly Epidemiol Rec 2017 92 19 241 268 28530369
  24. [26]Eddy DM Hollingworth W Caro JJ Tsevat J McDonald KM Wong JB; ISPOR−SMDM modeling good research practices task force. Model transparency and validation: a report of the ISPOR-SMDM modeling good research practices task Force-7 Value Health 2012 15 6 843 850 10.1016/j.jval.2012.04.012 22999134 · DOI · Paper page
  25. [27]Husereau D Drummond M Petrou S Greenberg D Augustovski F Briggs AH Consolidated health economic evaluation reporting standards (CHEERS) statement BMJ. 2013 346 f1049 10.1136/bmj.f1049 23529982 · DOI · Paper page
  26. [28]Whitty CJM What makes an academic paper useful for health policy? BMC Med 2015 13 301 10.1186/s12916-015-0544-8 26675206 PMC4682263 · DOI
  27. [29]Liberati A. Altman D. G Tetzlaff J. Mulrow C. Gotzsche P. C Ioannidis J. P A Clarke M. Devereaux P J Kleijnen J. Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration BMJ 2009 339 jul21 1 b2700 b2700 10.1136/bmj.b2700 19622552 PMC2714672 · DOI · Paper page
  28. [31]Ultsch B Damm O Beutels P Bilcke J Brüggenjürgen B Gerber-Grote A Methods for health economic evaluation of vaccines and immunization decision frameworks: a consensus framework from a European vaccine economics community Pharmacoeconomics. 2016 34 227 244 10.1007/s40273-015-0335-2 26477039 PMC4766233 · DOI
  29. [32]Eaton JW Johnson LF Salomon JA Bärnighausen T Bendavid E Bershteyn A HIV treatment as prevention: systematic comparison of mathematical models of the potential impact of antiretroviral therapy on HIV incidence in South Africa PLoS Med 2012 9 7 e1001245 10.1371/journal.pmed.1001245 22802730 PMC3393664 · DOI
  30. [33]Brisson M Bénard É Drolet M Bogaards JA Baussano L Vänskä S Population-level impact, herd immunity, and elimination after human papillomavirus vaccination: a systematic review and meta-analysis of predictions from transmission-dynamic models Lancet Public Health 2016 1 e8 17 10.1016/S2468-2667(16)30001-9 29253379 PMC6727207 · DOI · Paper page
  31. [35]Raftery AE Gneiting T Balabdaoui F Polakowski M Using Bayesian model averaging to calibrate forecast ensembles Mon Weather Rev 2005 133 5 1155 1174 10.1175/MWR2906.1 · DOI · Paper page
  32. [36]Park J Goldstein J Haran M Ferrari M An ensemble approach to predicting the impact of vaccination disease in Niger Vaccine. 2017 35 5835 5841 10.1016/j.vaccine.2017.09.020 28941619 PMC7185385 · DOI

Cited by

Sorted by citations

Authors
den Boon, S., Jit, M., Brisson, M., Medley, G., Beutels, P., White, R., Flasche, S., Hollingsworth, T. D., Garske, T., Pitzer, V. E., Hoogendoorn, M., Geffen, O., Clark, A., Kim, J., & Hutubessy, R.
Affiliations
  • Public Health England (GB)
  • Université Laval (CA)
  • London School of Hygiene & Tropical Medicine (GB)
  • University of Antwerp (BE)
  • University of Oxford (GB)
  • Imperial College London (GB)
  • Yale University (US)
  • Erasmus University Rotterdam (NL)
  • Harvard University (US)
  • World Health Organization (CH)
Corresponding author
Hutubessy, Raymond
ORCID
Venue
BMC Medicine
Publisher
Springer Science and Business Media LLC
Published
2019-08-19
Volume
17
Issue
1
Pages
163-163
ISSN
1741-7015 (ISSN-L)
DOI
10.1186/s12916-019-1403-9
PMC
PMC6699075
OpenAlex
W2968757301
Cited by
67
References
1
Citations by year
2026: 6 · 2025: 13 · 2024: 9 · 2023: 10 · 2022: 7 · 2021: 14 · 2020: 8
FWCI
9.05
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
  • LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Spiral (Imperial College London) (submittedVersion, other-oa)
Funders
  • Bill and Melinda Gates Foundation (OPP1151915, OPP1084276, OPP1160830, OPP1137034, OPP1135288, OPP1137034/OPP1151915)
  • World Health Organization
  • London School of Hygiene and Tropical Medicine
  • Wellcome Trust (Z/17/Z, 208812, 208812/Z/17/Z)
  • Royal Society (208812, Z/17/Z, 208812/Z/17/Z)
  • European Commission (EDCTP2-programme, EDCTP2, MR/P002404/1)
  • Department for International Development (MR/P002404/1)
  • Medical Research Council (MR/P002404/1, MR/R015600/1, OPP1135288, OPP1084276)
  • European and Developing Countries Clinical Trials Partnership (EDCTP2)
Topics
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Mathematical and Theoretical Epidemiology and Ecology Models
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
Keywords
multi-model comparison, model selection, system identification, vaccination effectiveness, cost-effectiveness, health policy decision-making, reporting guidelines
MeSH terms
Communicable Diseases, Cost-Benefit Analysis, Decision Making, Health Policy, Humans, Models, Theoretical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review