External Validity of Risk Models: Use of Benchmark Values to Disentangle a Case-Mix Effect From Incorrect Coefficients
Open accessAbstract
Various performance measures related to calibration and discrimination are available for the assessment of risk models. When the validity of a risk model is assessed in a new population, estimates of the model's performance can be influenced in several ways. The regression coefficients can be incorrect, which indeed results in an invalid model. However, the distribution of patient characteristics (case mix) may also influence the performance of the model. Here the authors consider a number of typical situations that can be encountered in external validation studies. Theoretical relations between differences in development and validation samples and performance measures are studied by simulation. Benchmark values for the performance measures are proposed to disentangle a case-mix effect from incorrect regression coefficients, when interpreting the model's estimated performance in validation samples. The authors demonstrate the use of the benchmark values using data on traumatic brain injury obtained from the International Tirilazad Trial and the North American Tirilazad Trial (1991-1994).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Interpretation of validation study results is often difficult because changes in prediction model performance are likely due to sampling error, differences in predictor effects, and/or differences in patient spectrum.
“Unfortunately, the interpretation of validation study results is often difficult, as changes in prediction model performance are likely to occur due to sampling error, differences in predictor effects, and/or differences in patient spectrum.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [10]
A model's discrimination performance tends to deteriorate when it is applied to populations or subgroups with a more homogeneous case-mix, as there is less separation of predicted risks across individuals.
“For instance, it is well known that a model’s discrimination performance tends to deteriorate when it is applied to populations or subgroups with a more homogeneous case-mix, as there is less separation of predicted risks across individuals.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [10]
An additional advantage of IPD is that it becomes possible to adjust discrimination performance for (variation in) subject-level covariates, and thus to gain more understanding in sources of between-study heterogeneity.
“An additional advantage of IPD is that it becomes possible to adjust discrimination performance for (variation in) subject-level covariates, and thus to gain more understanding in sources of between-study heterogeneity.9,10,26,68”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [10]
Common sources of heterogeneity are differences in study characteristics, differences in study quality, or differences in baseline risk across the validation studies.
“Common sources of heterogeneity are differences in study characteristics,62 differences in study quality, or differences in baseline risk across the validation studies.9,10”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305 · see citing works
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581 · see citing works
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270 · see citing works
A note on a general definition of the coefficient of determination
NAGELKERKE, N. J. D. (1991) · Biometrika · doi:10.1093/biomet/78.3.691 · Cited by 5,927 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Self-citationEvaluating the Yield of Medical Tests
Harrell, F. E. (1982) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1982.03320430047030 · Cited by 3,490 · see citing works
Gail, M. H., Brinton, L. A., Byar, D. P., Corle, D. K., Green, S. B., Schairer, C., & Mulvihill, J. J. (1989) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/81.24.1879 · Cited by 3,334 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Self-citationTime‐Dependent ROC Curves for Censored Survival Data and a Diagnostic Marker
Heagerty, P. J., Lumley, T., & PEPE, M. S. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00337.x · Cited by 2,870 · see citing works
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746 · see citing works
Self-citationD'Agostino, R. B., Grundy, S., Sullivan, L. M., Wilson, P., & for the CHD Risk Prediction Group (2001) · JAMA · doi:10.1001/jama.286.2.180 · Cited by 2,108 · see citing works
Survival Model Predictive Accuracy and ROC Curves
Heagerty, P. J., & Zheng, Y. (2005) · Biometrics · doi:10.1111/j.0006-341x.2005.030814.x · Cited by 1,471 · see citing works
Steyerberg, E. W., Mushkudiani, N. A., Perel, P., Butcher, I., Lü, J., McHugh, G. S., Murray, G. D., Marmarou, A., Roberts, I., Habbema, J. D. F., et al. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050165 · Cited by 1,437 · see citing works
Self-citationAssessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160 · see citing works
External validation is necessary in prediction research:
Bleeker, S. E., Moll, H. A., Steyerberg, E. W., Donders, A. R. T., Derksen-Lubsen, G., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00207-5 · Cited by 710 · see citing works
Self-citationVergouwe, Y., Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.06.017 · Cited by 636 · see citing works
Self-citationPredictive value of statistical models.
Van Houwelingen, J. C., & Le Cessie, S. (1990) · Statistics in Medicine · doi:10.1002/sim.4780091109 · Cited by 621 · see citing works
Regression, Prediction and Shrinkage
Copas, J. B. (1983) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1983.tb01258.x · Cited by 612 · see citing works
Statistical aspects of prognostic factor studies in oncology
Simon, R. M., & Altman, D. (1994) · British Journal of Cancer · doi:10.1038/bjc.1994.192 · Cited by 589 · see citing works
Regression models for prognostic prediction: advantages, problems, and suggested solutions.
Fe, H., Kl, L., Matchar, D. B., & Ta, R. (1985) · PubMed · Cited by 559 · see citing works
Two further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556 · see citing works
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552 · see citing works
Self-citationA multicenter trial on the efficacy of using tirilazad mesylate in cases of head injury
Marshall, L. F., Maas, A. I. R., Marshall, S. B., Bricolo, A., Fearnside, M. R., Iannotti, F., Klauber, M. R., Lagarrigue, J., Lobato, R., Persson, L., et al. (1998) · Journal of neurosurgery · doi:10.3171/jns.1998.89.4.0519 · Cited by 336 · see citing works
EXPLAINED VARIATION FOR LOGISTIC REGRESSION
Mittlböck, M., & SCHEMPER, M. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19961015)15:19 · Cited by 334 · see citing works
Validation techniques for logistic regression models
Miller, M. E., Hui, S. L., & Tierney, W. M. (1991) · Statistics in Medicine · doi:10.1002/sim.4780100805 · Cited by 322 · see citing works
The Performance of Risk Prediction Models
Gerds, T. A., Cai, T., & Schumacher, M. (2008) · Biometrical journal. Biometrische Zeitschrift · doi:10.1002/bimj.200810443 · Cited by 316 · see citing works
Systematic Review of Prognostic Models in Patients with Acute Stroke
Counsell, C. E., & Dennis, M. S. (2001) · Cerebrovascular Diseases · doi:10.1159/000047699 · Cited by 249 · see citing works
Validation of Biomarker-based risk prediction models
Taylor, J. M. G., Ankerst, D., & Andridge, R. (2008) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-07-4534 · Cited by 207 · see citing works
Mushkudiani, N. A., Hukkelhoven, C. W. P. M., Hernández, A. V., Murray, G. D., Choi, S. C., Maas, A. I. R., & Steyerberg, E. W. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.06.011 · Cited by 177 · see citing works
Self-citationHukkelhoven, C. W. P. M., Steyerberg, E. W., Farace, E., Habbema, J. D. F., Marshall, L. F., & Maas, A. I. R. (2002) · Journal of neurosurgery · doi:10.3171/jns.2002.97.3.0549 · Cited by 112 · see citing works
Self-citationStrategies for comparing treatments on a binary response with multi-centre data
Agresti, A., & Hartzel, J. S. (2000) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(20000430)19:8 · Cited by 102 · see citing works
Construction, validation and updating of a prognostic model for kidney graft survival
van Houwelingen, H. C., & Thorogood, J. (1995) · Statistics in Medicine · doi:10.1002/sim.4780141806 · Cited by 74 · see citing works
Contribution of three components to individual cancer risk predicting breast cancer risk in Italy
Boyle, P., Mezzetti, M., Vecchia, C. L., Franceschi, S., Decarli, A., & Robertson, C. (2004) · European Journal of Cancer Prevention · doi:10.1097/01.cej.0000130014.83901.53 · Cited by 44 · see citing works
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078
Hosmer (1997) · doi:10.1002/(sici)1097-0258(19970515)16:9
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 1×
“Furthermore, we estimated single overall performance measures to combine discrimination and calibration aspects, that is, R(2) and the Brier score (25).”
Flagged first, then by citations
Research works printed but not linked: 1
- Steyerberg EW, Eijkemans MJ, Harrell FE Jr, et al. Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med. 2000;19(8):1059-1079.
Cited by
Sorted by citations
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Self-citationMoons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018 · see citing works
Self-citationMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669 · see citing works
Self-citationCollins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665 · see citing works
A guide to systematic review and meta-analysis of prediction model performance
Debray, T. P. A., Damen, J. A. A. G., Snell, K. I. E., Ensor, J., Hooft, L., Reitsma, J. B., Riley, R. D., & Moons, K. G. M. (2017) · BMJ · doi:10.1136/bmj.i6460 · Cited by 657 · see citing works
Self-citationRiley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585 · see citing works
Self-citationLeening, M. J. G., Vedder, M. M., Witteman, J. C. M., Pencina, M. J., & Steyerberg, E. W. (2014) · Annals of Internal Medicine · doi:10.7326/m13-1522 · Cited by 594 · see citing works
Self-citationDebray, T. P. A., Vergouwe, Y., Koffijberg, H., Nieboer, D., Steyerberg, E. W., & Moons, K. G. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.06.018 · Cited by 597 · see citing works
Self-citationCalibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371 · see citing works
A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes
Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., Collins, G. S., & Moons, K. G. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218785504 · Cited by 278 · see citing works
“An additional advantage of IPD is that it becomes possible to adjust discrimination performance for (variation in) subject-level covariates, and thus to gain more understanding in sources of between-study heterogeneity.(9,10,26,68) Finally, methods for meta-analyzing clinical measures of performance such as net benefi…”
BackgroundMethodInfluentialSelf-citationAustin, P. C., & Steyerberg, E. W. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-82 · Cited by 270 · see citing works
Self-citationDamen, J. A., Pajouheshnia, R., Heus, P., Moons, K. G. M., Reitsma, J. B., Scholten, R. J. P. M., Hooft, L., & Debray, T. P. A. (2019) · BMC Medicine · doi:10.1186/s12916-019-1340-7 · Cited by 250 · see citing works
Self-citationSurveillance for ocular hypertension: an evidence synthesis and economic evaluation.
Burr, J., Botello-Pinzon, P., Takwoingi, Y., Hernández, R., Vazquez-Montes, M., Elders, A., Asaoka, R., Banister, K., van der Schoot, J., Fraser, C., et al. (2012) · Health Technology Assessment · doi:10.3310/hta16290 · Cited by 180 · see citing works
An examination of American College of Surgeons NSQIP surgical risk calculator accuracy.
Cohen, M. E., Liu, Y., Ko, C. Y., & Hall, B. L. (2017) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2016.12.057 · Cited by 176 · see citing works
Jenkins, D. A., Martin, G. P., Sperrin, M., Riley, R. D., Debray, T. P. A., Collins, G. S., & Peek, N. (2021) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00090-3 · Cited by 131 · see citing works
A new concordance measure for risk prediction models in external validation settings
van Klaveren, D., Gönen, M., Steyerberg, E. W., & Vergouwe, Y. (2016) · Statistics in Medicine · doi:10.1002/sim.6997 · Cited by 48 · see citing works
Self-citationJinks, R. C., Royston, P., & Parmar, M. K. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0078-y · Cited by 32 · see citing works
Calibration of clinical prediction rules does not just assess bias
Vach, W. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.003 · Cited by 26 · see citing works
Ajuwon, B. I., Richardson, A., Roper, K., & Lidbury, B. A. (2023) · Viruses · doi:10.3390/v15081735 · Cited by 6 · see citing works
- Authors
- Vergouwe, Y., Moons, K. G. M., & Steyerberg, E. W.
- Affiliations
- University Medical Center Utrecht (NL)
- Corresponding author
- Vergouwe, Y.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2010-08-31
- Volume
- 172
- Issue
- 8
- Pages
- 971-980
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- PMC
- PMC2984249
- OpenAlex
- W2104039704
- Cited by
- 243
- References
- 38
- Citations by year
- 2026: 15 · 2025: 11 · 2024: 13 · 2023: 13 · 2022: 21 · 2021: 24 · 2020: 16 · 2019: 20 · 2018: 17 · 2017: 15 · 2016: 15 · 2015: 14 · 2014: 21 · 2013: 11 · 2012: 12
- FWCI
- 7.62
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Journal of Epidemiology (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePub (Erasmus University Rotterdam) (submittedVersion)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (016.046.360)
- National Institute of Neurological Disorders and Stroke (NS-42691, R01NS042691)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- risk model, generalizability, external validation, benchmark values, calibration, discrimination (statistics), regression coefficients, traumatic brain injury, model performance measures
- MeSH terms
- Computer Simulation, Diagnosis-Related Groups, Humans, Risk Factors, Models, Statistical, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
