Validation of population-based disease simulation models: a review of concepts and methods
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Computer simulation models are used increasingly to support public health research and policy, but questions about their quality persist. The purpose of this article is to review the principles and methods for validation of population-based disease simulation models. METHODS: We developed a comprehensive framework for validating population-based chronic disease simulation models and used this framework in a review of published model validation guidelines. Based on the review, we formulated a set of recommendations for gathering evidence of model credibility. RESULTS: Evidence of model credibility derives from examining: 1) the process of model development, 2) the performance of a model, and 3) the quality of decisions based on the model. Many important issues in model validation are insufficiently addressed by current guidelines. These issues include a detailed evaluation of different data sources, graphical representation of models, computer programming, model calibration, between-model comparisons, sensitivity analysis, and predictive validity. The role of external data in model validation depends on the purpose of the model (e.g., decision analysis versus prediction). More research is needed on the methods of comparing the quality of decisions based on different models. CONCLUSION: As the role of simulation modeling in population health is increasing and models are becoming more complex, there is a need for further improvements in model validation methodology and common standards for evaluating model credibility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Factorial Sampling Plans for Preliminary Computational Experiments
Morris, M. D. (1991) · Technometrics · doi:10.1080/00401706.1991.10484804 · Cited by 4,943
Causal Diagrams for Epidemiologic Research
Greenland, S., Pearl, J., & Robins, J. M. (1999) · Epidemiology · doi:10.1097/00001648-199901000-00008 · Cited by 3,728
Weinstein, M. C., O'Brien, B., Hornberger, J., Jackson, J., Johannesson, M., McCabe, C., & Luce, B. R. (2003) · Value in Health · doi:10.1046/j.1524-4733.2003.00234.x · Cited by 1,172
Review of survival analyses published in cancer journals
Altman, D., De Stavola, B., Love, S., & Stepniewska, K. (1995) · British Journal of Cancer · doi:10.1038/bjc.1995.364 · Cited by 438
Screening for type 2 diabetes: literature review and economic modelling
Waugh, N., Scotland, G., McNamee, P., Gillett, M., Brennan, A., Goyder, E., Williams, R., & John, A. (2007) · Health Technology Assessment · doi:10.3310/hta11170 · Cited by 298
Recent developments in decision-analytic modelling for economic evaluation
Weinstein, M. C. (2006) · PharmacoEconomics · doi:10.2165/00019053-200624110-00002 · Cited by 220 · cited 1×
“It should be pointed out that model calibration methodology is evolving and there is no consensus at this time on how calibration should be carried out [36,37].”
BackgroundUncertainty and patient heterogeneity in medical decision models
Groot Koerkamp, B., Weinstein, M. C., Stijnen, T., Heijenbrok-Kal, M. H., & Hunink, M. G. M. (2010) · Medical Decision Making · doi:10.1177/0272989x09342277 · Cited by 105 · cited 1×
“Specifically, a stochastic rather than a deterministic model may be appropriate if modeling the full distribution of an outcome is important [39].”
Background
Cited by
Eddy, D. M., Hollingworth, W., Caro, J. J., Tsevat, J., McDonald, K. M., & Wong, J. B. (2012) · Medical Decision Making · doi:10.1177/0272989x12454579 · Cited by 799
Hayes, A. J., Leal, J., Gray, A. M., Holman, R. R., & Clarke, P. M. (2013) · Diabetologia · doi:10.1007/s00125-013-2940-y · Cited by 495
Eddy, D. M., Hollingworth, W., Caro, J. J., Tsevat, J., McDonald, K. M., & Wong, J. B. (2012) · Value in Health · doi:10.1016/j.jval.2012.04.012 · Cited by 466
Health Workforce Planning in OECD Countries: A Review of 26 Projection Models from 18 Countries
(2013) · OECD Health Working Papers · doi:10.1787/5k44t787zcwb-en · Cited by 286
Panayidou, K., Gsteiger, S., Egger, M., Kilcher, G., Carreras, M., Efthimiou, O., Debray, T. P. A., Trelle, S., Hummel, N., & on behalf of the GetReal methods review group (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1202 · Cited by 24
Metadata
- Authors
- Kopec, J. A., Finès, P., Manuel, D. G., Buckeridge, D. L., Flanagan, W. M., Oderkirk, J., Abrahamowicz, M., Harper, S., Sharif, B., Okhmatovskaia, A., Sayre, E. C., Rahman, M. M., & Wolfson, M. C.
- Venue
- BMC Public Health
- Published
- 2010-11-18
- Volume
- 10
- Issue
- 1
- Pages
- 710-710
- ISSN
- 1471-2458 (ISSN-L)
- OpenAlex
- W2071870246
- Cited by
- 122
- References
- 7
- FWCI
- 10.46
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.