Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist
Open accessThe DOI matches its Crossref record; 156 of the 157 references it cites were found in a registry or an index.
Abstract
Carl Moons and colleagues provide a checklist and background explanation for critically appraising and extracting data from systematic reviews of prognostic and diagnostic prediction modelling studies. Please see later in the article for the Editors' Summary.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Previous guidance for systematic reviews of prediction model studies mainly addressed formulation of the review aim, searching,12 critical appraisal (CHARMS)13 and data extraction of relevant studies. There is, however, little guidance on how to quantitatively synthesize the results of external validation studies.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Data extraction was conducted independently by 4 reviewers (R.Q., F.G, Y.A., and S.M.), following the recommendations in the CHARMS checklist.23”
From Linguistic markers for identifying post-traumatic stress disorder and associated symptoms: a systematic literature review (2025) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“Currently there is little information available to guide the development of a well-defined review question however the Quality in Prognosis Studies (QUIPS) tool [34] and the Checklist for critical appraisal and data extraction for systematic reviews of prediction modelling studies (CHARMS Checklist) [38] have been developed to assist in this process (see Table 1).”
From What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences (2018) · cites this paper as [38]
The cited source says what the claim attributes to it.
In the citing paper
“The Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) was employed to define the objectives of this systematic review and meta-analysis (29).”
From Machine learning-based risk predictive models for diabetic kidney disease in type 2 diabetes mellitus patients: a systematic review and meta-analysis (2025) · cites this paper as [29]
The cited source says what the claim attributes to it.
In the citing paper
“We employed the CHARMS checklist and the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines to develop standardized forms for data extraction (29, 32).”
From Machine learning-based risk predictive models for diabetic kidney disease in type 2 diabetes mellitus patients: a systematic review and meta-analysis (2025) · cites this paper as [29]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Self-citationvon Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00010 · Cited by 17,796
Self-citationQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Self-citation2013 ACCF/AHA Guideline for the Management of Heart Failure
Yancy, C. W., Jessup, M., Bozkurt, B., Butler, J., Casey, D. E., Drazner, M. H., Fonarow, G. C., Geraci, S. A., Horwich, T., Januzzi, J. L., et al. (2013) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.05.019 · Cited by 12,536
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349
Self-citationMultiple imputation using chained equations: Issues and guidance for practice
White, I. R., Royston, P., & Wood, A. M. (2011) · Statistics in Medicine · doi:10.1002/sim.4067 · Cited by 9,792 · cited 1×
“Numerous recommendations for reporting missing data in medical research have been proposed [103,104,114,115].”
BackgroundA simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655 · cited 1×
“The number of events-pervariable (EPV) is commonly used to calculate the sample size, where attaining a sample size with an EPV of ten or more is frequently recommended to avoid overfitting [97–101].”
MethodPrediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581 · cited 1×
“Prognostic models may involve models for individuals with a particular health condition, such as prediction of recurrence or death after diagnosis of breast cancer [8] or mortality after cardiac surgery [9], but also includes models for predicting the occurrence of future outcomes in apparently healthy individuals suc…”
BackgroundHarrell, F. E. (2001) · Springer Series in Statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,195
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
Self-citationSterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299 · cited 1×
“Numerous recommendations for reporting missing data in medical research have been proposed [103,104,114,115].”
BackgroundHarrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · cited 1×
“Reclassification measures, such as net reclassification improvement or index (NRI), evaluate whether a single biomarker has any incremental value to a prediction model [124,126].”
BackgroundDecision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · cited 1×
“Their use has been criticised as they rely on a priori–defined probability thresholds and do not account for difference in consequences of falsely reclassified individuals [122,127,128].”
BackgroundAssessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · cited 1×
“Their use has been criticised as they rely on a priori–defined probability thresholds and do not account for difference in consequences of falsely reclassified individuals [122,127,128].”
Background2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.005 · Cited by 4,945
Relaxing the Rule of Ten Events per Variable in Logistic and Cox Regression
Vittinghoff, E., & McCulloch, C. E. (2007) · American Journal of Epidemiology · doi:10.1093/aje/kwk052 · Cited by 4,359 · cited 1×
“The number of events-pervariable (EPV) is commonly used to calculate the sample size, where attaining a sample size with an EPV of ten or more is frequently recommended to avoid overfitting [97–101].”
MethodThe Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P. C., Lang, T., & for the CONSORT Group (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00012 · Cited by 3,835 · cited 2×
“Disappointingly, these key details are often poorly reported [72,75].”
BackgroundSelf-citationGoff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Circulation · doi:10.1161/01.cir.0000437741.48606.98 · Cited by 3,697
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Self-citationEuropean system for cardiac operative risk evaluation (EuroSCORE)
Nashef, S. A. M., Roques, F., Michel, P., Gauducheau, E., Lemeshow, S., & Salamon, R. (1999) · European Journal of Cardio-Thoracic Surgery · doi:10.1016/s1010-7940(99)00134-7 · Cited by 3,362 · cited 1×
“Prognostic models may involve models for individuals with a particular health condition, such as prediction of recurrence or death after diagnosis of breast cancer [8] or mortality after cardiac surgery [9], but also includes models for predicting the occurrence of future outcomes in apparently healthy individuals suc…”
BackgroundAssessing Bias in Studies of Prognostic Factors
Hayden, J. A., van der Windt, D. A., Cartwright, J. L., Côté, P., & Bombardier, C. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-4-201302190-00009 · Cited by 3,287 · cited 3×
“Existing guidance for synthesizing studies of prognostic factors [25,26] does not address studies of multivariable prediction models.”
BackgroundMethodClinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
The cost of dichotomising continuous variables
Altman, D. G., & Royston, P. (2006) · BMJ · doi:10.1136/bmj.332.7549.1080 · Cited by 2,688
Self-citationReview: a gentle introduction to imputation of missing values.
Donders, A. R. T., van der Heijden, G. J. M. G., Stijnen, T., & Moons, K. G. M. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.014 · Cited by 2,582
Self-citationExtensions of net reclassification improvement calculations to measure usefulness of new biomarkers
Pencina, M. J., D'Agostino, R. B., & Steyerberg, E. W. (2011) · Statistics in Medicine · doi:10.1002/sim.4085 · Cited by 2,453 · cited 1×
“Reclassification measures, such as net reclassification improvement or index (NRI), evaluate whether a single biomarker has any incremental value to a prediction model [124,126].”
BackgroundDichotomizing continuous predictors in multiple regression: a bad idea
Royston, P., Altman, D. G., & Sauerbrei, W. (2006) · Statistics in Medicine · doi:10.1002/sim.2331 · Cited by 2,312 · cited 2×
“, using restricted cubic splines or fractional polynomials) are indicated [45,51,58,93,94,96].”
BackgroundMethodSelf-citationPeduzzi, P., Concato, J., Feinstein, A. R., & Holford, T. R. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00048-8 · Cited by 2,154
Lindström, J., & Tuomilehto, J. (2003) · Diabetes Care · doi:10.2337/diacare.26.3.725 · Cited by 1,862 · cited 1×
“Prognostic models may involve models for individuals with a particular health condition, such as prediction of recurrence or death after diagnosis of breast cancer [8] or mortality after cardiac surgery [9], but also includes models for predicting the occurrence of future outcomes in apparently healthy individuals suc…”
BackgroundEmpirical evidence of design-related bias in studies of diagnostic tests.
Lijmer, J. G. (1999) · JAMA · doi:10.1001/jama.282.11.1061 · Cited by 1,861 · cited 1×
“, blinded to) the results of the index tests is well established [35,40,54,69,79,80,85].”
BackgroundMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.004 · Cited by 1,858
Self-citationPrognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · cited 3×
“Regardless of the modelling technique, bootstrapping is particularly recommended in small datasets with many candidate predictors and when predictor selection techniques have been used [3,49,51, 55,58,59].”
BackgroundMethodSelf-citationSteyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491
Evaluation of the Quality of Prognosis Studies in Systematic Reviews
Hayden, J. A., Côté, P., & Bombardier, C. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-6-200603210-00010 · Cited by 1,452 · cited 3×
“Existing guidance for synthesizing studies of prognostic factors [25,26] does not address studies of multivariable prediction models.”
BackgroundMethodPrognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436 · cited 10×
“, primary or secondary care), and number of centres, is important to allow for proper assessment of the applicability and thus generalizability of the study findings [40,41,56,72].”
BackgroundInfluentialSelf-citationREporting recommendations for tumour MARKer prognostic studies (REMARK).
for the Statistics Subcommittee of the NCI-EORTC Working Group on Cancer Diagnostics, McShane, L. M., Altman, D. G., Sauerbrei, W., Taube, S. E., Gion, M., & Clark, G. M. (2005) · British Journal of Cancer · doi:10.1038/sj.bjc.6602678 · Cited by 1,394
Self-citationClinical Prediction Rules: Applications and Methodological Standards
Wasson, J. H., Sox, H. C., Neff, R. K., & Goldman, L. (1985) · New England Journal of Medicine · doi:10.1056/nejm198509263131306 · Cited by 1,357
Eurp-pean system for cardiac operative risk evaluation (Euro SCORE)
Nashef, S. A. M. (1999) · Medical Entomology and Zoology · Cited by 1,349
Prognosis and prognostic research: what, why, and how?
Moons, K. G. M., Royston, P., Vergouwe, Y., Grobbee, D. E., & Altman, D. G. (2009) · BMJ · doi:10.1136/bmj.b375 · Cited by 1,337 · cited 5×
“For example, relevance to physicians and patients is enhanced by models that predict patient-relevant outcomes, such as death, pain, or recurrence of disease, rather than those that predict process outcomes such as duration of hospital stay or intermediate outcomes, except when there is a clear and established causal…”
BackgroundInfluentialSelf-citationRisk prediction models: II. External validation, model updating, and impact assessment
Moons, K. G. M., Kengne, A. P., Grobbee, D. E., Royston, P., Vergouwe, Y., Altman, D. G., & Woodward, M. (2012) · Heart · doi:10.1136/heartjnl-2011-301247 · Cited by 1,195 · cited 6×
“Hence, if relevant, the systematic review should report the performance measures of the original and ‘‘rounded’’ models where this information is available in the published primary report [32].”
BackgroundMethodInfluentialSelf-citationPrognosis and prognostic research: Developing a prognostic model
Royston, P., Moons, K. G. M., Altman, D. G., & Vergouwe, Y. (2009) · BMJ · doi:10.1136/bmj.b604 · Cited by 1,172 · cited 3×
“In both cases, overfitting may arise, particularly in small datasets [51,55,58,59].”
BackgroundMethodSelf-citationAssessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160
The STARD Statement for Reporting Studies of Diagnostic Accuracy: Explanation and Elaboration
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Moher, D., Rennie, D., de Vet, H. C. W., & Lijmer, J. G. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00012-w1 · Cited by 1,124 · cited 1×
“, blinded to) the results of the index tests is well established [35,40,54,69,79,80,85].”
BackgroundSelf-citationThe Nottingham prognostic index in primary breast cancer
Galea, M. H., Blamey, R. W., Elston, C. E., & Ellis, I. O. (1992) · Breast Cancer Research and Treatment · doi:10.1007/bf01840834 · Cited by 1,070 · cited 1×
“Prognostic models may involve models for individuals with a particular health condition, such as prediction of recurrence or death after diagnosis of breast cancer [8] or mortality after cardiac surgery [9], but also includes models for predicting the occurrence of future outcomes in apparently healthy individuals suc…”
BackgroundUsers' Guides to the Medical Literature: XXII: How to Use Articles About Clinical Decision Rules
McGinn, T. G., Guyatt, G. H., Wyer, P. C., Naylor, C. D., Stiell, I. G., Richardson, W. S., & for the Evidence-Based Medicine Working Group (2000) · JAMA · doi:10.1001/jama.284.1.79 · Cited by 1,014
Using the outcome for imputation of missing predictor values was preferred
Moons, K. G. M., Donders, R. A. R. T., Stijnen, T., & Harrell, F. E. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.009 · Cited by 994
Self-citationRegression with Missing X 's: A Review
Little, R. J. A. (1992) · Journal of the American Statistical Association · doi:10.1080/01621459.1992.10476282 · Cited by 968
Prognosis and prognostic research: application and impact of prognostic models in clinical practice
Moons, K. G. M., Altman, D. G., Vergouwe, Y., & Royston, P. (2009) · BMJ · doi:10.1136/bmj.b606 · Cited by 934 · cited 1×
“, primary care versus secondary care), in the age range of participants (children versus adults), in the clinical inclusion criteria, or even by using different predictor or outcome definitions and measurements [1,31,32,56,129,132].”
BackgroundSelf-citationInappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis
Sun, G.-W., Shook, T. L., & Kay, G. L. (1996) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(96)00025-x · Cited by 933 · cited 1×
“Although common, such screening or pre-selection based on univariable significance testing carries a great risk of so-called predictor selection bias [51,56,58,116].”
BackgroundExternal validation of a Cox prognostic model: principles and methods
Royston, P., & Altman, D. G. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-33 · Cited by 911
Self-citationMoons, K. G. M., Kengne, A. P., Woodward, M., Royston, P., Vergouwe, Y., Altman, D. G., & Grobbee, D. E. (2012) · Heart · doi:10.1136/heartjnl-2011-301246 · Cited by 889
Self-citationSources of Variation and Bias in Studies of Diagnostic Accuracy: A Systematic Review
Whiting, P., Rutjes, A. W. S., Reitsma, J. B., Glas, A. S., Bossuyt, P. M. M., & Kleijnen, J. (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-140-3-200402030-00010 · Cited by 798 · cited 1×
“, blinded to) the results of the index tests is well established [35,40,54,69,79,80,85].”
BackgroundSelf-citationConcato, J., Peduzzi, P., Holford, T. R., & Feinstein, A. R. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00510-2 · Cited by 797 · cited 1×
“The number of events-pervariable (EPV) is commonly used to calculate the sample size, where attaining a sample size with an EPV of ten or more is frequently recommended to avoid overfitting [97–101].”
MethodReilly, B. M., & Evans, A. T. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-3-200602070-00009 · Cited by 794 · cited 4×
“Several strategies exist depending on the availability of data, but are broadly categorised as internal and external validation [1,32,49,51,56,58,59,129].”
BackgroundInfluentialLaupacis, A. (1997) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1997.03540300056034 · Cited by 793 · cited 2×
“The same issue is also important in prognostic studies where the assessor of the outcome occurrence should be blinded to ascertainment of the predictor [47,48].”
BackgroundMethodSteyerberg, E. W., Eijkemans, M. J. C., Harrell, F. E., & Habbema, J. D. F. (2000) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(20000430)19:8<1059::aid-sim412>3.0.co;2-0 · Cited by 785
Systematic reviews of evaluations of prognostic variables
Altman, D. G. (2001) · BMJ · doi:10.1136/bmj.323.7306.224 · Cited by 756 · cited 3×
“Different definitions and measurement methods of candidate predictors are a potential source of heterogeneity and thus risk of bias, and the use of different measurement methods may affect the strength of predictors and influence whether the predictors ultimately are included in the prediction model [42,61].”
Self-citationStatistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC Curve
Cook, N. R. (2008) · Clinical Chemistry · doi:10.1373/clinchem.2007.096529 · Cited by 691 · cited 1×
“The c-statistic should not be used as the only performance measure, however, since it is influenced by the distribution of predictor values and is often insensitive to inclusion of an additional predictor in the model [59,122–125].”
Vergouwe, Y., Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.06.017 · Cited by 636 · cited 1×
“For studies validating prediction models, sample size considerations are not well established, but a minimum of 100 events and 100 non-events have been suggested [102].”
BackgroundSelf-citationPredictive value of statistical models.
Van Houwelingen, J. C., & Le Cessie, S. (1990) · Statistics in Medicine · doi:10.1002/sim.4780091109 · Cited by 621 · cited 1×
“The need for use of shrinkage methods increases with smaller datasets, although in datasets with a low number of EPV, even shrinkage methods cannot account for all bias [51,58,117,118].”
MethodEvidence of bias and variation in diagnostic accuracy studies
Rutjes, A. W. S., Reitsma, J. B., Di Nisio, M., Smidt, N., van Rijn, J. C., & Bossuyt, P. M. M. (2006) · Canadian Medical Association Journal · doi:10.1503/cmaj.050090 · Cited by 611 · cited 1×
“, blinded to) the results of the index tests is well established [35,40,54,69,79,80,85].”
BackgroundSelf-citationReporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600 · cited 5×
“Continuous or categorical predictors are frequently dichotomised for the analysis [2,42,56,60,78] despite strong evidence and recommendations to the contrary [92–94].”
BackgroundMethodInfluentialSelf-citationMissing covariate data in medical research: To impute is better than to ignore
Janssen, K. J. M., Donders, A. R. T., Harrell, F. E., Vergouwe, Y., Chen, Q., Grobbee, D. E., & Moons, K. G. M. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.12.008 · Cited by 592 · cited 1×
“However, reporting on the frequency and type of missing data is often poor [2,15,56,62,78,103–105] despite the adverse effects of missing data on development, validation, and updating of a prediction model [34,103,105–112].”
BackgroundSelf-citationValidation, updating and impact of clinical prediction rules: A review
Toll, D. B., Janssen, K. J. M., Vergouwe, Y., & Moons, K. G. M. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.04.008 · Cited by 580
Self-citationMethodologic Standards for the Development of Clinical Decision Rules in Emergency Medicine
STIELL, I., & WELLS, G. (1999) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(99)70309-4 · Cited by 571 · cited 1×
“A review of 83 diagnostic prediction models for detection of ovarian malignancy found that studies often sampled participants non-consecutively [70], increasing the risk of bias due to selective sampling [42,56,71].”
BackgroundSteyerberg, E. W., Eijkemans, M. J. C., Harrell, F. E., & Habbema, J. D. F. (2001) · Medical Decision Making · doi:10.1177/0272989x0102100106 · Cited by 550 · cited 1×
“However, such tests have frequently been criticised because of the limited statistical power to assess poor calibration and being oversensitive in large samples [58,117,120,121].”
BackgroundEarly Prediction of Outcome of Activities of Daily Living After Stroke: A Systematic Review
Veerbeek, J. M., Kwakkel, G., van Wegen, E. E. H., Ket, J. C. F., & Heymans, M. W. (2011) · Stroke · doi:10.1161/strokeaha.110.604090 · Cited by 537 · cited 1×
“N Prognostic models for activities of daily living, to be used in the early post-stroke phase [46].”
MethodStiell, I. G. (1993) · JAMA · doi:10.1001/jama.269.9.1127 · Cited by 537
Evaluation of diagnostic tests when there is no gold standard. A review of methods
Rutjes, A., Reitsma, J., Coomarasamy, A., Khan, K., & Bossuyt, P. (2007) · Health Technology Assessment · doi:10.3310/hta11500 · Cited by 524 · cited 2×
“This is often used in diagnostic studies for target diseases where the reference standard used in practice is known to include a subjective assessment of information [52,54,88,89].”
BackgroundMethodSelf-citationCase–Control and Two-Gate Designs in Diagnostic Accuracy Studies
Rutjes, A. W., Reitsma, J. B., Vandenbroucke, J. P., Glas, A. S., & Bossuyt, P. M. (2005) · Clinical Chemistry · doi:10.1373/clinchem.2005.048595 · Cited by 501
Self-citationHow should variable selection be performed with multiply imputed data?
Wood, A. M., White, I. R., & Royston, P. (2008) · Statistics in Medicine · doi:10.1002/sim.3177 · Cited by 485 · cited 1×
“These adverse effects are related to the amount of missing data [112] and the extent to which data are missing completely at random [108,111].”
BackgroundCollins, G. S., Mallett, S., Omar, O., & Yu, L.-M. (2011) · BMC Medicine · doi:10.1186/1741-7015-9-103 · Cited by 484 · cited 7×
“Numerous systematic reviews of prediction models have reported that the number of events per candidate predictor is often poorly reported and, when it is reported, that the EPV is often less than ten [2,15,56,78].”
BackgroundInfluentialSelf-citationFerreira-González, I., Permanyer-Miralda, G., Domingo-Salvany, A., Busse, J. W., Heels-Ansdell, D., Montori, V. M., Akl, E. A., Bryant, D. M., Alonso-Coello, P., Alonso, J., et al. (2007) · BMJ · doi:10.1136/bmj.39136.682083.ae · Cited by 473 · cited 1×
“combined outcomes is particularly challenging [82,83].”
Bennette, C., & Vickers, A. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-21 · Cited by 444 · cited 1×
“Continuous or categorical predictors are frequently dichotomised for the analysis [2,42,56,60,78] despite strong evidence and recommendations to the contrary [92–94].”
BackgroundPeters, S. A. E., den Ruijter, H. M., Bots, M. L., & Moons, K. G. M. (2012) · Heart · doi:10.1136/heartjnl-2011-300747 · Cited by 415 · cited 1×
“N Added predictive value of carotid imaging markers to existing cardiovascular predictors in the general population [140].”
BackgroundSelf-citationReitsma, J. B., Rutjes, A. W. S., Khan, K. S., Coomarasamy, A., & Bossuyt, P. M. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.02.005 · Cited by 405 · cited 2×
“This is often used in diagnostic studies for target diseases where the reference standard used in practice is known to include a subjective assessment of information [52,54,88,89].”
BackgroundMethodSelf-citationUpdating methods improved the performance of a clinical prediction model in new patients
Janssen, K. J. M., Moons, K. G. M., Kalkman, C. J., Grobbee, D. E., & Vergouwe, Y. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.04.018 · Cited by 374 · cited 2×
“When a review includes both development and validation studies of the same model(s), or several external validations of the same model, reporting differences in frequency (binary) and distribution (continuous) of the predictors and outcomes across the study samples is recommended, as a different case-mix is known to r…”
BackgroundSelf-citationGeersing, G.-J., Bouwmeester, W., Zuithoff, P., Spijker, R., Leeflang, M., & Moons, K. (2012) · PLoS ONE · doi:10.1371/journal.pone.0032844 · Cited by 350 · cited 1×
“Since then, search strategies for identifying prognostic and diagnostic prediction model studies have been developed [21–23], validated, and further refined [24].”
MethodSelf-citationLeeflang, M. M. G., Moons, K. G. M., Reitsma, J. B., & Zwinderman, A. H. (2008) · Clinical Chemistry · doi:10.1373/clinchem.2007.096032 · Cited by 346 · cited 1×
“The reporting of performance based on thresholds chosen from the data itself can produce over-optimistic and biased performance [95].”
BackgroundSelf-citationSystematic review of prognostic models in traumatic brain injury
Perel, P., Edwards, P., Wentz, R., & Roberts, I. (2006) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-6-38 · Cited by 344 · cited 4×
“There are over 100 models for predicting outcome after brain trauma [12], over 60 models for breast cancer prognosis [13], 45 models for cardiovascular events after being diagnosed with diabetes [14], 43 models for predicting prevalent and incident type 2 diabetes [15], and 20 models for predicting prolonged intensive…”
BackgroundMethodInfluentialTime-dependent bias was common in survival analyses published in leading clinical journals
van Walraven, C., Davis, D., Forster, A. J., & Wells, G. A. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2003.12.008 · Cited by 337
Courvoisier, D. S., Combescure, C., Agoritsas, T., Gayet-Ageron, A., & Perneger, T. V. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.11.012 · Cited by 320 · cited 1×
“The number of events-pervariable (EPV) is commonly used to calculate the sample size, where attaining a sample size with an EPV of ten or more is frequently recommended to avoid overfitting [97–101].”
MethodComparisons of established risk prediction models for cardiovascular disease: systematic review
Siontis, G. C. M., Tzoulaki, I., Siontis, K. C., & Ioannidis, J. P. A. (2012) · BMJ · doi:10.1136/bmj.e3318 · Cited by 300 · cited 2×
“N Relative predictive performance of specific prognostic models for occurrence of cardiovascular disease when applied in general populations [44].”
BackgroundClinical Epidemiology: Principles, Methods, And Applications For Clinical Research
Grobbee, D. E., & Hoes, A. W. (2008) · Cited by 289
Development and validation of a prediction model with missing predictor data: a practical approach
Vergouwe, Y., Royston, P., Moons, K. G. M., & Altman, D. G. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.03.017 · Cited by 287 · cited 1×
“These adverse effects are related to the amount of missing data [112] and the extent to which data are missing completely at random [108,111].”
BackgroundSelf-citationAssessment of Claims of Improved Prediction Beyond the Framingham Risk Score
Tzoulaki, I. (2009) · JAMA · doi:10.1001/jama.2009.1757 · Cited by 276 · cited 2×
“Review of added value of specific predictor or updating of a specific model (development and/or validation) N Added predictive value of C-reactive protein to the Framingham risk scores [134].”
BackgroundMethodMethodological challenges in the evaluation of prognostic factors in breast cancer.
Altman, D. G., & Lyman, G. H. (1998) · Breast Cancer Research and Treatment · doi:10.1023/a:1006193704132 · Cited by 271 · cited 1×
“For reasons described above, data driven methods to create risk groups, such as the ‘‘optimal’’ probability threshold method or at the median, are not recommended [135].”
MethodSelf-citationValidation, calibration, revision and combination of prognostic survival models
van Houwelingen, H. C. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24<3401::aid-sim554>3.0.co;2-2 · Cited by 268
Systematic Review of Prognostic Models in Patients with Acute Stroke
Counsell, C., & Dennis, M. (2001) · Cerebrovascular Diseases · doi:10.1159/000047699 · Cited by 250 · cited 3×
“Reviews of prediction models for specific outcomes in a target population (development and/or validation) N Prognostic models for survival, for independence in activities of daily living, and for getting home, in patients with acute stroke [27].”
BackgroundMethodMarshall, A., Altman, D. G., Royston, P., & Holder, R. L. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-7 · Cited by 247 · cited 2×
“These adverse effects are related to the amount of missing data [112] and the extent to which data are missing completely at random [108,111].”
BackgroundSelf-citationCollins, G. S., & Altman, D. G. (2012) · BMJ · doi:10.1136/bmj.e4181 · Cited by 244 · cited 1×
“The predictive accuracy of models may change over time and require periodic updating [76], as was done for the QRISK models [77].”
MethodSelf-citationVergouwe, Y., Moons, K. G. M., & Steyerberg, E. W. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq223 · Cited by 243 · cited 1×
“Quantifying model performance in other individuals is often referred to as model validation (Box 1) [1,32,49,51,56,58,59, 119,129,130].”
BackgroundSelf-citationThe use and reporting of multiple imputation in medical research – a review
Mackinnon, A. (2010) · Journal of Internal Medicine · doi:10.1111/j.1365-2796.2010.02274.x · Cited by 229 · cited 1×
“Numerous recommendations for reporting missing data in medical research have been proposed [103,104,114,115].”
BackgroundBurton, A., & Altman, D. G. (2004) · British Journal of Cancer · doi:10.1038/sj.bjc.6601907 · Cited by 219 · cited 1×
“Numerous recommendations for reporting missing data in medical research have been proposed [103,104,114,115].”
van Dieren, S., Beulens, J. W. J., Kengne, A. P., Peelen, L. M., Rutten, G. E. H. M., Woodward, M., van der Schouw, Y. T., & Moons, K. G. M. (2012) · Heart · doi:10.1136/heartjnl-2011-300734 · Cited by 216 · cited 2×
“There are over 100 models for predicting outcome after brain trauma [12], over 60 models for breast cancer prognosis [13], 45 models for cardiovascular events after being diagnosed with diabetes [14], 43 models for predicting prevalent and incident type 2 diabetes [15], and 20 models for predicting prolonged intensive…”
BackgroundSelf-citationReporting methods in studies developing prognostic models in cancer: a review
Mallett, S., Royston, P., Dutton, S., Waters, R., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-20 · Cited by 206 · cited 4×
“Recent systematic reviews have found the reporting of performance measures to be poor, with reliance on measures of discrimination [2,15,60].”
BackgroundMethodInfluentialSelf-citationReporting performance of prognostic models in cancer: a review.
Mallett, S., Royston, P., Waters, R., Dutton, S., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-21 · Cited by 195 · cited 3×
“Numerous systematic reviews of prediction models have reported that the number of events per candidate predictor is often poorly reported and, when it is reported, that the EPV is often less than ten [2,15,56,78].”
BackgroundSelf-citationCollins, G. S., Omar, O., Shanyinde, M., & Yu, L.-M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.06.020 · Cited by 192 · cited 2×
“However, reporting on the frequency and type of missing data is often poor [2,15,56,62,78,103–105] despite the adverse effects of missing data on development, validation, and updating of a prediction model [34,103,105–112].”
BackgroundMethodThe Accuracy of Risk Scores in Predicting Ovarian Malignancy: A Systematic Review
Geomini, P., Kruitwagen, R., Bremer, G. L., Cnossen, J., & Mol, B. W. J. (2009) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e318195ad17 · Cited by 184 · cited 2×
“Several reviews have reported that many studies did not blind the outcome measurement for the candidate predictors [70,81,86,87].”
BackgroundSearching for Clinical Prediction Rules in MEDLINE
Ingui, B. J., & Rogers, M. A. M. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080391 · Cited by 184
Prediction models in reproductive medicine: a critical appraisal†
Leushuis, E., van der Steeg, J. W., Steures, P., Bossuyt, P. M. M., Eijkemans, M. J. C., van der Veen, F., Mol, B. W. J., & Hompes, P. G. A. (2009) · Human Reproduction Update · doi:10.1093/humupd/dmp013 · Cited by 173 · cited 3×
“Reviews of all existing models in a particular clinical field (development and/or validation) N Existing prognostic models in reproductive medicine [29].”
BackgroundMethodRietveld, R. P., Riet, G. T., Bindels, P. J. E., Sloos, J. H., & van Weert, H. C. P. M. (2004) · BMJ · doi:10.1136/bmj.38128.631319.ae · Cited by 168 · cited 1×
“Diagnostic models are aimed at calculating the probability that an individual has a certain disorder, such as deep vein thrombosis [4,5], ankle fractures [6], or conjunctivitis [7].”
BackgroundDealing with Missing Predictor Values When Applying Clinical Prediction Models
Janssen, K. J. M., Vergouwe, Y., Donders, A. R. T., Harrell, F. E., Chen, Q., Grobbee, D. E., & Moons, K. G. M. (2009) · Clinical Chemistry · doi:10.1373/clinchem.2008.115345 · Cited by 151 · cited 1×
“However, reporting on the frequency and type of missing data is often poor [2,15,56,62,78,103–105] despite the adverse effects of missing data on development, validation, and updating of a prediction model [34,103,105–112].”
BackgroundSelf-citationDecision rules for the use of radiography in acute ankle injuries
Graebe, A. (1993) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(05)80138-6 · Cited by 151
Moons, K. G. M., Hoes, A. W., & Oudega, R. (2005) · Thrombosis and Haemostasis · doi:10.1160/th04-12-0829 · Cited by 143 · cited 1×
“Diagnostic models are aimed at calculating the probability that an individual has a certain disorder, such as deep vein thrombosis [4,5], ankle fractures [6], or conjunctivitis [7].”
BackgroundSelf-citationWells, Hirsh, Anderson, Lensing, Foster, Kearon, Weitz, D'ovidio, Cogo, Prandoni, et al. (1998) · Journal of Internal Medicine · doi:10.1046/j.1365-2796.1998.00249.x · Cited by 142 · cited 1×
“Diagnostic models are aimed at calculating the probability that an individual has a certain disorder, such as deep vein thrombosis [4,5], ankle fractures [6], or conjunctivitis [7].”
BackgroundAdvantages of the nested case-control design in diagnostic research
Biesheuvel, C. J., Vergouwe, Y., Oudega, R., Hoes, A. W., Grobbee, D. E., & Moons, K. G. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-48 · Cited by 135
Self-citationShould a Head-Injured Child Receive a Head CT Scan? A Systematic Review of Clinical Prediction Rules
Maguire, J. L., Boutis, K., Uleryk, E. M., Laupacis, A., & Parkin, P. C. (2009) · Pediatrics · doi:10.1542/peds.2009-0075 · Cited by 134
Performance of the original EuroSCORE
Siregar, S., Groenwold, R. H. H., de Heer, F., Bots, M. L., van der Graaf, Y., & van Herwerden, L. A. (2012) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezr285 · Cited by 131 · cited 2×
“Review of the performance of one or more specific models (validation) N Predictive performance of the EuroSCORE for operative mortality following cardiac surgery when validated in other patient samples [139].”
BackgroundMethodEttema, R. G. A., Peelen, L. M., Schuurmans, M. J., Nierich, A. P., Kalkman, C. J., & Moons, K. G. M. (2010) · Circulation · doi:10.1161/circulationaha.109.926808 · Cited by 130 · cited 1×
“There are over 100 models for predicting outcome after brain trauma [12], over 60 models for breast cancer prognosis [13], 45 models for cardiovascular events after being diagnosed with diabetes [14], 43 models for predicting prevalent and incident type 2 diabetes [15], and 20 models for predicting prolonged intensive…”
BackgroundSelf-citationAssessing New Biomarkers and Predictive Models for Use in Clinical Practice: A Clinician's Guide
McGeechan, K., Macaskill, P., Irwig, L., Liew, G., & Wong, T. Y. (2008) · Archives of Internal Medicine · doi:10.1001/archinte.168.21.2304 · Cited by 129 · cited 1×
“Finally, we identified key methodological literature discussing recommended approaches for the design, conduct, analysis, and reporting of prediction models, followed by a search of the corresponding reference lists [3,19,31,32,37,47–59].”
MethodPrognostic Models: A Methodological Framework and Review of Models for Breast Cancer
Altman, D. G. (2009) · Cancer Investigation · doi:10.1080/07357900802572110 · Cited by 129
Self-citationSerrano, L. A., Hess, E. P., Bellolio, M. F., Murad, M. H., Montori, V. M., Erwin, P. J., & Decker, W. W. (2010) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2010.05.013 · Cited by 129 · cited 1×
“Several reviews have reported that many studies did not blind the outcome measurement for the candidate predictors [70,81,86,87].”
BackgroundMallett, S., Timmer, A., Sauerbrei, W., & Altman, D. G. (2010) · British Journal of Cancer · doi:10.1038/sj.bjc.6605462 · Cited by 122 · cited 2×
“The checklist is designed to help form a review question for and appraisal of all types of primary prediction modelling studies, including, regressions, neural network, genetic programming, and vector machine learning models [1– 3,12,14,15,27–30].”
BackgroundMethodSelf-citationDeveloping optimal search strategies for detecting sound clinical prediction studies in MEDLINE.
Wong, S. S.-L., Wilczynski, N. L., Haynes, R. B., & Ramkissoonsingh, R. (2003) · PubMed Central · Cited by 115
Everything you always wanted to know about evaluating prediction models (but were too afraid to ask)
Vickers, A. J., & Cronin, A. M. (2010) · Urology · doi:10.1016/j.urology.2010.06.019 · Cited by 112 · cited 1×
“Prognostic models commonly have a better predictive accuracy for short-term outcomes than for long-term outcomes (item 6) [63].”
BackgroundComparing risk prediction models
Collins, G. S., & Moons, K. G. M. (2012) · BMJ · doi:10.1136/bmj.e3186 · Cited by 110 · cited 1×
“Furthermore, one may like to overview the performance of all prediction models for a specific outcome or target population before making decisions on which model to apply in routine practice [138].”
BackgroundSelf-citationTamariz, L. J., Eng, J., Segal, J. B., Krishnan, J. A., Bolger, D. T., Streiff, M. B., Jenckes, M. W., & Bass, E. B. (2004) · The American Journal of Medicine · doi:10.1016/j.amjmed.2004.04.021 · Cited by 110 · cited 1×
“N Prediction models for diagnosis of venous thromboembolism in patients suspected of having the disease [28].”
MethodBias arising from missing data in predictive models.
Gorelick, M. H. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.11.029 · Cited by 107 · cited 1×
“, missing data is still partly due to unobserved data or characteristics of the participants, multiple imputation does not sufficiently solve the invalidity problem [107,112,113].”
BackgroundPepe, M. S., Feng, Z., & Gu, J. W. (2007) · Statistics in Medicine · doi:10.1002/sim.2991 · Cited by 107
Problems With Risk Reclassification Methods for Evaluating Prediction Models
Pepe, M. S. (2011) · American Journal of Epidemiology · doi:10.1093/aje/kwr013 · Cited by 101
Kyzas, P. A., Denaxa-Kyza, D., & Ioannidis, J. P. A. (2007) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djk032 · Cited by 94 · cited 1×
“We then reviewed published systematic reviews of prediction models and prognostic factor studies, along with the checklists or quality appraisal criteria used in those reviews [12,27–29,42–46].”
MethodPeek, N., Arts, D. G. T., Bosman, R. J., van der Voort, P. H. J., & de Keizer, N. F. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.08.011 · Cited by 92 · cited 1×
“However, such tests have frequently been criticised because of the limited statistical power to assess poor calibration and being oversensitive in large samples [58,117,120,121].”
BackgroundClinical Prediction Rules for Physical Therapy Interventions: A Systematic Review
Beneciuk, J. M., Bishop, M. D., & George, S. Z. (2009) · Physical Therapy · doi:10.2522/ptj.20080239 · Cited by 91 · cited 2×
“Disappointingly, these key details are often poorly reported [72,75].”
BackgroundSome prognostic models for traumatic brain injury were not valid
Hukkelhoven, C. W. P. M., Rampen, A. J. J., Maas, A. I. R., Farace, E., Habbema, J. D. F., Marmarou, A., Marshall, L. F., Murray, G. D., & Steyerberg, E. W. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.06.009 · Cited by 90 · cited 1×
“When a review includes both development and validation studies of the same model(s), or several external validations of the same model, reporting differences in frequency (binary) and distribution (continuous) of the predictors and outcomes across the study samples is recommended, as a different case-mix is known to r…”
BackgroundGanna, A., Reilly, M., de Faire, U., Pedersen, N., Magnusson, P., & Ingelsson, E. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kwr374 · Cited by 89 · cited 2×
“Data from cohort, nested case-control, or case-cohort studies are recommended for prognostic model development and validation studies, and cross-sectional designs for diagnostic modelling studies [47,58,59,65–67].”
BackgroundMethodInventory of prostate cancer predictive tools
Shariat, S. F., Karakiewicz, P. I., Margulis, V., & Kattan, M. W. (2008) · Current Opinion in Urology · doi:10.1097/mou.0b013e3282f9b3e5 · Cited by 82 · cited 2×
“, using restricted cubic splines or fractional polynomials) are indicated [45,51,58,93,94,96].”
MethodOudega, R., Moons, K. G. M., & Hoes, A. W. (2004) · Family Practice · doi:10.1093/fampra/cmh718 · Cited by 77 · cited 1×
“For example, prediction models developed in secondary care perform less well when evaluated in a primary care setting [73,74].”
BackgroundRiley, R. D., Ridley, G. F., Williams, K. J., Altman, D. G., Hayden, J. A., & de Vet, H. C. W. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.02.004 · Cited by 75 · cited 1×
“To evaluate the proliferation of prediction models, systematic reviews are necessary and led to the formation of the Cochrane Collaboration Prognosis Reviews Methods Group [19,20].”
MethodSelf-citationDiagnostic studies as multivariable, prediction research
Moons, K. G. M., & Grobbee, D. E. (2002) · Journal of Epidemiology and Community Health · doi:10.1136/jech.56.5.337 · Cited by 75
Self-citationWhen should we remain blind and when should our eyes remain open in diagnostic studies?
Moons, K. G. M., & Grobbee, D. E. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(02)00408-0 · Cited by 75 · cited 1×
“For example, if the predictive ability of an MRI in addition to laboratory measurements is studied, the MRI should be interpreted blinded to the laboratory measurements to reduce possible bias [73,91].”
BackgroundSelf-citationMethods to evaluate risks for composite end points and their individual components
Glynn, R. J., & ROSNER, B. A. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2003.02.001 · Cited by 74 · cited 1×
“combined outcomes is particularly challenging [82,83].”
Risk assessment of fragility fractures: summary of NICE guidance
Rabar, S., Lau, R., O'Flynn, N., Li, L., Barry, P., & on behalf of the Guideline Development Group (2012) · BMJ · doi:10.1136/bmj.e3698 · Cited by 74 · cited 1×
“Furthermore, prediction models are increasingly being appraised and recommended for formal risk assessment in clinical guidelines [17,18].”
BackgroundHess, E. P., Thiruganasambandamoorthy, V., Wells, G. A., Erwin, P. J., Jaffe, A. S., Hollander, J. E., Montori, V. M., & Stiell, I. G. (2008) · Canadian Journal of Emergency Medicine · doi:10.1017/s148180350001040x · Cited by 73 · cited 1×
“Several reviews have reported that many studies did not blind the outcome measurement for the candidate predictors [70,81,86,87].”
BackgroundMinne, L., Eslami, S., de Keizer, N., de Jonge, E., de Rooij, S. E., & Abu-Hanna, A. (2012) · Intensive Care Medicine · doi:10.1007/s00134-011-2390-2 · Cited by 69
Prognosis research: Why is Dr. Lydgate still waiting?
Hemingway, H. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.02.005 · Cited by 68 · cited 1×
“To evaluate the proliferation of prediction models, systematic reviews are necessary and led to the formation of the Cochrane Collaboration Prognosis Reviews Methods Group [19,20].”
MethodDiagnostic research on routine care data
Oostenbrink, R., Moons, K. G. M., Bleeker, S. E., Moll, H. A., & Grobbee, D. E. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00080-5 · Cited by 64
Self-citationAre standardized mortality ratios valid for public health data analysis?
Goldman, D. A., & Brender, J. D. (2000) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(20000430)19:8<1081::aid-sim406>3.0.co;2-a · Cited by 49
Vergouwe, Y., Steyerberg, E. W., de Wit, R., Roberts, J. T., Keizer, H. J., Collette, L., Stenning, S. P., & Habbema, J. D. F. (2003) · British Journal of Cancer · doi:10.1038/sj.bjc.6600759 · Cited by 44 · cited 1×
“When a review includes both development and validation studies of the same model(s), or several external validations of the same model, reporting differences in frequency (binary) and distribution (continuous) of the predictors and outcomes across the study samples is recommended, as a different case-mix is known to r…”
BackgroundSelf-citationModeling the effects of continuous risk factors.
Royston, P., Sauerbrei, W., & Altman, D. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00163-8 · Cited by 44 · cited 1×
“, using restricted cubic splines or fractional polynomials) are indicated [45,51,58,93,94,96].”
MethodKeogh, C., Wallace, E., O’Brien, K. K., Murphy, P., Teljeur, C., McGrath, B., Smith, S. M., Doherty, N., Dimitrov, B. D., & Fahey, T. P. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.11.011 · Cited by 38 · cited 1×
“Since then, search strategies for identifying prognostic and diagnostic prediction model studies have been developed [21–23], validated, and further refined [24].”
MethodStrengthening the reporting of genetic risk prediction studies (GRIPS): explanation and elaboration
Janssens, A. C. J., Ioannidis, J. P., Bedrosian, S., Boffetta, P., Dolan, S. M., Dowling, N., Fortier, I., Freedman, A. N., Grimshaw, J. M., Gulcher, J., et al. (2011) · European Journal of Human Genetics · doi:10.1038/ejhg.2011.27 · Cited by 34
Discrepancies in predicted fracture risk in elderly people
Bolland, M. J., Jackson, R., Gamble, G. D., & Grey, A. B. (2013) · BMJ · doi:10.1136/bmj.e8669 · Cited by 30
Challenges in dia-prognostic research
Knottnerus, J. A. (2002) · Journal of Epidemiology & Community Health · doi:10.1136/jech.56.5.340 · Cited by 25
van Zaane, B., Vergouwe, Y., Donders, A. R. T., & Moons, K. G. M. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-166 · Cited by 21
Self-citationStrengthening the reporting of genetic risk prediction studies (GRIPS): explanation and elaboration
Janssens, A. C. J. W., Ioannidis, J. P. A., Bedrosian, S., Boffetta, P., Dolan, S. M., Dowling, N., Fortier, I., Freedman, A. N., Grimshaw, J. M., Gulcher, J., et al. (2011) · European Journal of Epidemiology · doi:10.1007/s10654-011-9551-z · Cited by 17 · cited 3×
“Finally, we identified key methodological literature discussing recommended approaches for the design, conduct, analysis, and reporting of prediction models, followed by a search of the corresponding reference lists [3,19,31,32,37,47–59].”
MethodJacob, M., Bruegger, D., Conzen, P., Becker, B. F., FINSTERER, U., & Rehm, M. (2005) · Transfusion · doi:10.1111/j.0041-1132.2005.04292.x · Cited by 17 · cited 3×
“Different definitions and measurement methods of candidate predictors are a potential source of heterogeneity and thus risk of bias, and the use of different measurement methods may affect the strength of predictors and influence whether the predictors ultimately are included in the prediction model [42,61].”
BackgroundMethodPrognostic Models: A Methodological Framework and Review of Models for Breast Cancer
Altman, D. G. (2007) · CRC Press eBooks · doi:10.3109/9781420019940.002 · Cited by 15
Self-citationBreast Cancer: Translational Therapeutic Strategies
(2007) · CRC Press eBooks · doi:10.3109/9781420019940 · Cited by 9
Gondrie, M. J. A., Janssen, K. J. M., Moons, K. G. M., & van der Graaf, Y. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.01.021 · Cited by 7 · cited 1×
“predictive accuracy of the model in the validation dataset [84].”
MethodSelf-citationDetection of broadband signals in signal-dependent noise
Poor, H. V. (1990) · The Journal of the Acoustical Society of America · doi:10.1121/1.398797 · Cited by 4
What do we mean by validating a prognostic model?
(2000) · doi:10.1002/(sici)1097-0258(20000229)19:4<453::aid-sim350>3.0.co;2-5
Self-citationHarrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4<361::aid-sim168>3.0.co;2-4
Cited by
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
“The TRIPOD Statement does not prescribe how studies developing, validating, or updating prediction models should be undertaken, nor should it be used as a tool for explicitly assessing quality or quantifying risk of bias in such studies (52).”
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
“The TRIPOD Statement does not prescribe how studies developing, validating, or updating prediction models should be undertaken, nor should it be used as a tool for explicitly assessing quality or quantifying risk of bias in such studies [52].”
MethodSelf-citationWynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Self-citation
Metadata
- Authors
- Moons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S.
- Affiliations
- University Medical Center Utrecht (NL)
- University of Oxford (GB)
- Corresponding author
- Moons, Karel G. M.
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-10-14
- Volume
- 11
- Issue
- 10
- Pages
- e1001744
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC4196729
- OpenAlex
- W2077663753
- Semantic Scholar
- c79bfa9a67c8745d7620fcfe8cf582b9d0407073
- Cited by
- 2,018
- References
- 157
- Citations by year
- 2026: 319 · 2025: 348 · 2024: 253 · 2023: 229 · 2022: 214 · 2021: 219 · 2020: 144 · 2019: 93 · 2018: 74 · 2017: 57 · 2016: 35 · 2015: 28 · 2014: 1
- FWCI
- 11.19
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, cc-by)
- Funders
- Cancer Research UK (16895)
- National Institute for Health and Care Research (NF-SI-0513-10131)
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (project 918.10.615, 918.10.615)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- critical appraisal, data extraction, predictive modeling, prognostic model, diagnostic model, systematic reviews, reporting quality
- MeSH terms
- Decision Support Techniques, Humans, Models, Theoretical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.