A new framework to enhance the interpretation of external validation studies of clinical prediction models
Open accessAbstract
OBJECTIVES: It is widely acknowledged that the performance of diagnostic and prognostic prediction models should be assessed in external validation studies with independent data from "different but related" samples as compared with that of the development sample. We developed a framework of methodological steps and statistical methods for analyzing and enhancing the interpretation of results from external validation studies of prediction models. STUDY DESIGN AND SETTING: We propose to quantify the degree of relatedness between development and validation samples on a scale ranging from reproducibility to transportability by evaluating their corresponding case-mix differences. We subsequently assess the models' performance in the validation sample and interpret the performance in view of the case-mix differences. Finally, we may adjust the model to the validation setting. RESULTS: We illustrate this three-step framework with a prediction model for diagnosing deep venous thrombosis using three validation samples with varying case mix. While one external validation sample merely assessed the model's reproducibility, two other samples rather assessed model transportability. The performance in all validation samples was adequate, and the model did not require extensive updating to correct for miscalibration or poor fit to the validation settings. CONCLUSION: The proposed framework enhances the interpretation of findings at external validation of prediction models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Interpretation of validation study results is often difficult because changes in prediction model performance are likely due to sampling error, differences in predictor effects, and/or differences in patient spectrum.
“Unfortunately, the interpretation of validation study results is often difficult, as changes in prediction model performance are likely to occur due to sampling error, differences in predictor effects, and/or differences in patient spectrum.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [9]
A model's discrimination performance tends to deteriorate when it is applied to populations or subgroups with a more homogeneous case-mix, as there is less separation of predicted risks across individuals.
“For instance, it is well known that a model’s discrimination performance tends to deteriorate when it is applied to populations or subgroups with a more homogeneous case-mix, as there is less separation of predicted risks across individuals.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [9]
An additional advantage of IPD is that it becomes possible to adjust discrimination performance for (variation in) subject-level covariates, and thus to gain more understanding in sources of between-study heterogeneity.
“An additional advantage of IPD is that it becomes possible to adjust discrimination performance for (variation in) subject-level covariates, and thus to gain more understanding in sources of between-study heterogeneity.9,10,26,68”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [9]
Common sources of heterogeneity are differences in study characteristics, differences in study quality, or differences in baseline risk across the validation studies.
“Common sources of heterogeneity are differences in study characteristics,62 differences in study quality, or differences in baseline risk across the validation studies.9,10”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371 · see citing works
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581 · see citing works
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,357 · see citing works
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group (2010) · BMJ · doi:10.1136/bmj.c332 · Cited by 9,329 · see citing works
An Essay concerning Human Understanding
Locke, J. (1690) · Oxford University Press eBooks · doi:10.1093/oseo/instance.00018020 · Cited by 5,201 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Self-citationGail, M. H., Brinton, L. A., Byar, D. P., Corle, D. K., Green, S. B., Schairer, C., & Mulvihill, J. J. (1989) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/81.24.1879 · Cited by 3,334 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Self-citationRao, S. J. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s263 · Cited by 1,986 · see citing works
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597 · see citing works
Clinical Prediction Models—a Practical Approach to Development, Validation and Updating
Bedogni, G. (2009) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2009.00614_16.x · Cited by 1,526 · see citing works
Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · see citing works
Self-citationWhat's In A Name? Malay Seals As Onomastic Sources
Gallop, A. T. (2018) · Malay Literature · doi:10.37052/ml.31(1)no1 · Cited by 1,495 · see citing works
Prognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436 · see citing works
Self-citationPrognosis and prognostic research: what, why, and how?
Moons, K. G. M., Royston, P., Vergouwe, Y., Grobbee, D. E., & Altman, D. G. (2009) · BMJ · doi:10.1136/bmj.b375 · Cited by 1,337 · see citing works
Self-citationRisk prediction models: II. External validation, model updating, and impact assessment
Moons, K. G. M., Kengne, A. P., Grobbee, D. E., Royston, P., Vergouwe, Y., Altman, D. G., & Woodward, M. (2012) · Heart · doi:10.1136/heartjnl-2011-301247 · Cited by 1,195 · see citing works
Self-citationAssessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160 · see citing works
Prognosis and prognostic research: application and impact of prognostic models in clinical practice
Moons, K. G. M., Altman, D. G., Vergouwe, Y., & Royston, P. (2009) · BMJ · doi:10.1136/bmj.b606 · Cited by 934 · see citing works
Self-citationMoons, K. G. M., Kengne, A. P., Woodward, M., Royston, P., Vergouwe, Y., Altman, D. G., & Grobbee, D. E. (2012) · Heart · doi:10.1136/heartjnl-2011-301246 · Cited by 889 · see citing works
Self-citationReilly, B. M., & Evans, A. T. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-3-200602070-00009 · Cited by 794 · see citing works
Collins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729 · see citing works
Self-citationThe Concept of External Validity
Calder, B. J., Phillips, L. W., & Tybout, A. M. (1982) · Journal of Consumer Research · doi:10.1086/208920 · Cited by 620 · see citing works
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600 · cited 1× · see citing works
“Differences in study-level characteristics (such as inclusion and exclusion criteria, details on subject recruitment, or study design choices) may provide additional insights into achieved model performance and could therefore further improve the interpretation of prediction model validation results (step 3) [47].”
Self-citationValidation, updating and impact of clinical prediction rules: A review
Toll, D. B., Janssen, K. J. M., Vergouwe, Y., & Moons, K. G. M. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.04.008 · Cited by 584 · see citing works
Self-citationTwo further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556 · see citing works
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552 · see citing works
Self-citationUpdating methods improved the performance of a clinical prediction model in new patients
Janssen, K. J. M., Moons, K. G. M., Kalkman, C. J., Grobbee, D. E., & Vergouwe, Y. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.04.018 · Cited by 381 · see citing works
Self-citationTheory-Testing, Generalization, and the Problem of External Validity
Lucas, J. W. (2003) · Sociological Theory · doi:10.1111/1467-9558.00187 · Cited by 374 · see citing works
Commentary: Prognostic models: clinically useful or quickly forgotten?
Wyatt, J. C., & Altman, D. G. (1995) · BMJ · doi:10.1136/bmj.311.7019.1539 · Cited by 367 · see citing works
Prediction of outcome after moderate and severe traumatic brain injury
Roozenbeek, B., Lingsma, H. F., Lecky, F. E., Lü, J., Weir, J., Butcher, I., McHugh, G. S., Murray, G. D., Perel, P., Maas, A. I. R., et al. (2012) · Critical Care Medicine · doi:10.1097/ccm.0b013e31824519ce · Cited by 292 · see citing works
Self-citationVergouwe, Y., Moons, K. G. M., & Steyerberg, E. W. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq223 · Cited by 243 · see citing works
Self-citationOn criteria for evaluating models of absolute risk
Gail, M. H. (2005) · Biostatistics · doi:10.1093/biostatistics/kxi005 · Cited by 231 · see citing works
Geersing, G., Zuithoff, N. P. A., Kearon, C., Anderson, D. R., ten Cate‐Hoek, A. J., Elf, J. L., Bates, S. M., Hoes, A. W., Kraaijenhagen, R. A., Oudega, R., et al. (2014) · BMJ · doi:10.1136/bmj.g1340 · Cited by 227 · see citing works
Self-citationDebray, T. P. A., Moons, K. G. M., Ahmed, I., Koffijberg, H., & Riley, R. D. (2013) · Statistics in Medicine · doi:10.1002/sim.5732 · Cited by 215 · see citing works
Self-citationGross, C. P., Mallory, R. M., Heiat, A., & Krumholz, H. M. (2002) · Annals of Internal Medicine · doi:10.7326/0003-4819-137-1-200207020-00007 · Cited by 186 · see citing works
The Area under the ROC Curve and Its Competitors
Hilden, J. (1991) · Medical Decision Making · doi:10.1177/0272989x9101100204 · Cited by 183 · see citing works
(2002) · Annals of Internal Medicine · doi:10.7326/0003-4819-137-1-200207020-00002 · Cited by 172 · see citing works
Overinterpretation and Misreporting of Diagnostic Accuracy Studies: Evidence of “Spin”
Ochodo, E. A., de Haan, M. C., Reitsma, J. B., Hooft, L., Bossuyt, P. M., & Leeflang, M. M. G. (2013) · Radiology · doi:10.1148/radiol.12120527 · Cited by 158 · see citing works
The Wells Rule Does Not Adequately Rule Out Deep Venous Thrombosis in Primary Care Patients
Oudega, R., Hoes, A. W., & Moons, K. G. M. (2005) · Annals of Internal Medicine · doi:10.7326/0003-4819-143-2-200507190-00008 · Cited by 152 · see citing works
Self-citationSpectrum bias--why clinicians need to be cautious when applying diagnostic test studies
Willis, B. H. (2008) · Family Practice · doi:10.1093/fampra/cmn051 · Cited by 148 · see citing works
Validation of probabilistic predictions
Miller, M. E., Langefeld, C. D., Tierney, W. M., Hui, S. L., & McDonald, C. J. (1993) · Medical Decision Making · doi:10.1177/0272989x9301300107 · Cited by 129 · see citing works
A simple method to adjust clinical prediction models to local circumstances
Janssen, K. J. M., Vergouwe, Y., Kalkman, C. J., Grobbee, D. E., & Moons, K. G. M. (2009) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/s12630-009-9041-x · Cited by 126 · see citing works
Self-citationValidity of prognostic models: When is a model clinically useful?
Vergouwe, Y., Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2002) · Seminars in Urologic Oncology · doi:10.1053/suro.2002.32521 · Cited by 123 · see citing works
Self-citationOdds Ratios and Logistic Regression: Further Examples of their use and Interpretation
Hailpern, S. M., & VISINTAINER, P. F. (2003) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0300300301 · Cited by 96 · see citing works
POPPERIAN REFUTATION IN EPIDEMIOLOGY
Maclure, M. (1985) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a114005 · Cited by 88 · see citing works
A nonparametric measure of the overlapping coefficient
Clemons, T. E., & Bradley, E. L. (2000) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(99)00074-2 · Cited by 87 · see citing works
Comparing Two Samples: Extensions of the t , Rank-Sum, and Log-Rank Tests
O’Brien, P. C. (1988) · Journal of the American Statistical Association · doi:10.1080/01621459.1988.10478564 · Cited by 84 · see citing works
Variation in baseline risk as an explanation of heterogeneity in meta-analysis
Walter, S. D. (1997) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19971230)16:24 · Cited by 80 · see citing works
Collins, G. S., & Altman, D. (2012) · British Journal of Cancer · doi:10.1038/bjc.2012.266 · Cited by 60 · see citing works
DIAGNOSTIC PREDICTION RULES: PRINCIPLES, REQUIREMENTS, AND PITFALLS
Knottnerus, J. A. (1995) · Primary Care: Clinics in Office Practice · doi:10.1016/s0095-4543(21)00522-4 · Cited by 56 · see citing works
Mizuno, S., Yamaguchi, T., Fukushima, A., et al. (2005) · Clinical Trials · doi:10.1191/1740774505cn077oa · Cited by 54 · see citing works
Comparing Two Samples: Extensions of the t, Rank-Sum, and Log-Rank Tests
O’Brien, P. C. (1988) · Journal of the American Statistical Association · doi:10.2307/2288918 · Cited by 50 · see citing works
A generalized Mahalanobis distance for mixed data
de Leon, A. R., & Carriere, K. C. (2003) · Journal of Multivariate Analysis · doi:10.1016/j.jmva.2003.08.006 · Cited by 47 · see citing works
Toll, D. B., Oudega, R., Vergouwe, Y., Moons, K. G. M., & Hoes, A. W. (2007) · Family Practice · doi:10.1093/fampra/cmm075 · Cited by 22 · see citing works
Self-citationTransparency and Reproducible Research in Modeling
Smith-Spangler, C. M. (2012) · Medical Decision Making · doi:10.1177/0272989x12458977 · Cited by 14 · see citing works
Knottnerus, J. A. (1992) · Medical Decision Making · doi:10.1177/0272989x9201200407 · Cited by 5 · see citing works
Hahn, S., Williamson, P. R., Hutton, J. L., Garner, P., & Flynn, E. V. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24
Flagged first, then by citations
Research works printed but not linked: 2
- Miller ME, Langefeld CD, Tierney WM, Hui SL, McDonald CJ. Vali- dation of probabilistic predictions. Med Decis Making 1993;13:49e58.
- van Houwelingen HC. Validation, calibration, revision and combi- nation of prognostic survival models. Stat Med 2000;19:3401e15.
Cited by
Sorted by citations
Radiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688 · see citing works
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Self-citationKey challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136 · see citing works
Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669 · see citing works
Self-citationPrediction models for cardiovascular disease risk in the general population: systematic review
Damen, J. A. A. G., Hooft, L., Schuit, E., Debray, T. P. A., Collins, G. S., Tzoulaki, I., Lassale, C. M., Siontis, G. C. M., Chiocchia, V., Roberts, C., et al. (2016) · BMJ · doi:10.1136/bmj.i2416 · Cited by 935 · see citing works
Self-citationExternal validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887 · see citing works
Evaluation of clinical prediction models (part 1): from development to external validation
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-074819 · Cited by 798 · see citing works
Self-citationCollins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665 · see citing works
A guide to systematic review and meta-analysis of prediction model performance
Debray, T. P. A., Damen, J. A. A. G., Snell, K. I. E., Ensor, J., Hooft, L., Reitsma, J. B., Riley, R. D., & Moons, K. G. M. (2017) · BMJ · doi:10.1136/bmj.i6460 · Cited by 657 · see citing works
Self-citationRiley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585 · see citing works
Self-citationDeveloping clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
Self-citationMinimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400 · see citing works
Self-citationCalibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312 · see citing works
A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes
Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., Collins, G. S., & Moons, K. G. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218785504 · Cited by 278 · see citing works
“An additional advantage of IPD is that it becomes possible to adjust discrimination performance for (variation in) subject-level covariates, and thus to gain more understanding in sources of between-study heterogeneity.(9,10,26,68) Finally, methods for meta-analyzing clinical measures of performance such as net benefi…”
BackgroundMethodInfluentialSelf-citationBinuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01801-8 · Cited by 246 · see citing works
Self-citationDamen, J. A., Pajouheshnia, R., Heus, P., Moons, K. G. M., Reitsma, J. B., Scholten, R. J. P. M., Hooft, L., & Debray, T. P. A. (2019) · BMC Medicine · doi:10.1186/s12916-019-1340-7 · Cited by 250 · see citing works
“Hence, one external validation may not be sufficient to claim adequate performance and multiple validations are necessary to get insight in the generalizability of prediction models [71].”
BackgroundSelf-citationEvaluating the impact of prediction models: lessons learned, challenges, and recommendations
Kappen, T. H., van Klei, W. A., van Wolfswinkel, L., Kalkman, C. J., Vergouwe, Y., & Moons, K. G. M. (2018) · Diagnostic and Prognostic Research · doi:10.1186/s41512-018-0033-6 · Cited by 215 · see citing works
Self-citationMeehan, A. J., Lewis, S. J., Fazel, S., Fusar-Poli, P., Steyerberg, E. W., Stahl, D., & Danese, A. (2022) · Molecular Psychiatry · doi:10.1038/s41380-022-01528-4 · Cited by 219 · see citing works
Self-citationMinimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198 · see citing works
The Science of Prognosis in Psychiatry: A Review
Fusar-Poli, P., Hijazi, Z., Stahl, D., & Steyerberg, E. W. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.2530 · Cited by 197 · see citing works
Self-citationJolani, S., Debray, T. P. A., Koffijberg, H., van Buuren, S., & Moons, K. G. M. (2015) · Statistics in Medicine · doi:10.1002/sim.6451 · Cited by 192 · see citing works
Self-citationAn examination of American College of Surgeons NSQIP surgical risk calculator accuracy.
Cohen, M. E., Liu, Y., Ko, C. Y., & Hall, B. L. (2017) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2016.12.057 · Cited by 176 · see citing works
Song, X., Yu, A. S. L., Kellum, J. A., Waitman, L. R., Matheny, M. E., Simpson, S. Q., Hu, Y., & Liu, M. (2020) · Nature Communications · doi:10.1038/s41467-020-19551-w · Cited by 120 · see citing works
Fusar-Poli, P., Werbeloff, N., Rutigliano, G., Oliver, D., Davies, C., Stahl, D., McGuire, P., & Osborn, D. (2019) · Schizophrenia Bulletin · doi:10.1093/schbul/sby070 · Cited by 95 · see citing works
Li, K., O'Brien, R., Lutz, M., Luo, S., & Initiative, T. A. D. N. (2018) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2017.11.004 · Cited by 44 · see citing works
Postpartum depression: a developed and validated model predicting individual risk in new mothers
Munk-Olsen, T., Liu, X., Madsen, K. B., Kjeldsen, M.-M. Z., Petersen, L. V., Bergink, V., Skalkidou, A., Vigod, S. N., Frokjaer, V. G., Pedersen, C. B., et al. (2022) · Translational Psychiatry · doi:10.1038/s41398-022-02190-8 · Cited by 36 · see citing works
Puntis, S., Oliver, D., & Fusar-Poli, P. (2021) · Schizophrenia Research · doi:10.1016/j.schres.2021.01.005 · Cited by 21 · see citing works
Instability of the AUROC of clinical prediction models
van Leeuwen, F. D., Steyerberg, E. W., van Klaveren, D., Wessler, B., Kent, D. M., & van Zwet, E. W. (2025) · Statistics in Medicine · doi:10.1002/sim.70011 · Cited by 19 · see citing works
Self-citationPrediction models in gynaecology: Transparent reporting needed for clinical application
Gelderblom, M. E., Stevens, K. Y. R., Houterman, S., Weyers, S., & Schoot, B. C. (2021) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/j.ejogrb.2021.08.013 · Cited by 11 · see citing works
Sharafutdinov, K., Bhat, J. S., Fritsch, S. J., Nikulina, K., E. Samadi, M., Polzin, R., Mayer, H., Marx, G., Bickenbach, J., & Schuppert, A. (2022) · Frontiers in Big Data · doi:10.3389/fdata.2022.603429 · Cited by 10 · see citing works
van Slobbe, R., Herrmannova, D., Boeke, D. J., Lima-Walton, E. S., Abu-Hanna, A., & Vagliano, I. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105815 · Cited by 7 · see citing works
Ajuwon, B. I., Richardson, A., Roper, K., & Lidbury, B. A. (2023) · Viruses · doi:10.3390/v15081735 · Cited by 6 · see citing works
He, T. T., Jiao, T. Q., & An, X. M. (2026) · PeerJ · doi:10.7717/peerj.20770 · Cited by 2 · see citing works
- Authors
- Debray, T. P. A., Vergouwe, Y., Koffijberg, H., Nieboer, D., Steyerberg, E. W., & Moons, K. G. M.
- Affiliations
- University Medical Center Utrecht (NL)
- Erasmus MC (NL)
- Corresponding author
- Debray, Thomas P.A.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2015-03
- Volume
- 68
- Issue
- 3
- Pages
- 279-289
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W1997192661
- Cited by
- 597
- References
- 58
- Citations by year
- 2026: 84 · 2025: 44 · 2024: 54 · 2023: 55 · 2022: 74 · 2021: 79 · 2020: 58 · 2019: 41 · 2018: 35 · 2017: 32 · 2016: 22 · 2015: 17 · 2014: 1
- FWCI
- 25.34
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-sa)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- University of Twente Research Information (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- external validation, clinical prediction model, model portability, model reproducibility, model updating, model calibration, prognostic model, diagnostic model
- MeSH terms
- Data Interpretation, Statistical, Female, Humans, Male, Predictive Value of Tests, Reproducibility of Results, Models, Statistical, Outcome Assessment, Health Care, Validation Studies as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
