A calibration hierarchy for risk models was defined: from utopia to empirical data
Open accessAbstract
Objective Calibrated risk models are vital for valid decision support. We define four levels of calibration and describe implications for model development and external validation of predictions.Study design and setting We present results based on simulated data sets.Results A common definition of calibration is "having an event rate of R% among patients with a predicted risk of R%," which we refer to as "moderate calibration." Weaker forms of calibration only require the average predicted risk (mean calibration) or the average prediction effects (weak calibration) to be correct. "Strong calibration" requires that the event rate equals the predicted risk for every covariate pattern. This implies that the model is fully correct for the validation setting. We argue that this is unrealistic: the model type may be incorrect, the linear predictor is only asymptotically unbiased, and all nonlinear and interaction effects should be correctly modeled. In addition, we prove that moderate calibration guarantees nonharmful decision making. Finally, results indicate that a flexible assessment of calibration in small validation data sets is problematic.Conclusion Strong calibration is desirable for individualized decision support but unrealistic and counter productive by stimulating the development of overly complex models. Model development and external validation should focus on moderate calibration.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 17 cited sources checked · 0 supported · 0 flagged
- Findingown data
There was some miscalibration in mean risk for Examples 1-2 where the development sample was small, even though the correct underlying model formulation was estimated.
“There was some miscalibration in mean risk for Examples 1-2 where the development sample was small, even though the correct underlying model formulation was estimated.”
Quote in the paperNot a substantial claim - Findingown data
Enlarging the development dataset alleviated the overfitting and calibration problems.
“Enlarging the development dataset alleviated the problems (Example 4 in Table 1, Figure 1D).”
Quote in the paperSubstantial claim - Findingown data
The flexible calibration curve is more variable and hence requires more data than the calibration slope for a stable assessment.
“The results suggest that the flexible calibration curve is more variable and hence requires more data than the calibration slope for a stable assessment.”
Quote in the paperSubstantial claim - Findingown data
Observed calibration curves will easily deviate from the diagonal line even when the model is moderately or strongly calibrated.
“It is clear that observed calibration curves will easily deviate from the diagonal line even when the model is moderately or strongly calibrated.”
Quote in the paperSubstantial claim - Findingreasoning
The decision curve for the true model is the best one in Example 9.
“Nevertheless, the decision curve for the true model is the best one.”
Quote in the paperSubstantial claim
- Findingown data
In Example 7, only the flexible calibration curve reveals that calibration is problematic.
“Again, only the flexible calibration curve reveals that calibration is problematic.”
Quote in the paperNot a substantial claim - Findingreasoning
The moderately recalibrated version of Example 9 has Net Benefit at least as high as that of the default strategies.
“The moderately recalibrated version does not have that problem anymore: Net Benefit is now at least as high as that of the default strategies.”
Quote in the paperSubstantial claim - Findingown data
In Example 9, the original miscalibrated model has Net Benefit below that of treat all or treat none for a subset of risk thresholds.
“The original model has Net Benefit below that of treat all or treat none for a subset of risk thresholds (Figure 5).”
Quote in the paperNot a substantial claim - Findingown data
For larger development datasets, the disagreement between observed and predicted risks disappeared.
“For larger development datasets the disagreement between observed and predicted risks disappeared (Examples 3-5).”
Quote in the paperNot a substantial claim - Findingown data
Flexible assessment of calibration in small validation datasets is problematic.
“results indicate that a flexible assessment of calibration in small validation datasets is problematic.”
- Also stated in: § 2.2. Illustration: Examples 1-5 For illustration, we consider five simulated examples, as previously presented [5].We randomly generate four independent predictor variables to .These predictor variables are ordinal with three categories (-1, 0, and 1) that each have 33% prevalence in order to visualize calibration by covariate pattern.Let outcome be generated by an underlying logistic regression model with the true linear predictor , hence the intercept equals 0, and to are assumed to have linear main effects [5].In simulations, the true probability of event is calculated using the linear predictor and the observed outcome is generated as a Bernoulli variable from .Then, we develop a model to predict based on to , which means that we are using the correct model formulation.Finally, we validate on a new dataset that is generated with the exact same procedure.We repeat this process four times to illustrate the influence of random variability: (1) using 100 simulated patients for development and 100 for validation, (2) using 100 for development and 10,000 for validation, (3) using 10,000 for development and 100 for validation, (4) using 10,000 for development and 10,000 for validation, and (5) using 10 million for development and 10 million for validation.Estimation of model coefficients was unstable when the development data contains only 100 patients (Table 1), despite having more than 10 events per variable (4 parameters, 44 non-events, 56 events).Risk estimates were overfitted, as evidenced by the calibration slope and calibration curves at validation (Example 2 in Table 1, and Figure 1B) as well as the calibration curves.Further, the calibration intercept was negative which is suggestive of general overestimation.Enlarging the development dataset alleviated the problems (Example 4 in Table 1, Figure 1D).Similar issues emerge when using a small validation dataset (Figures 1A and 1C.Calibration results for Examples 1 and 3 are unstable, in particular the flexible calibration curve.With 10 million patients the coefficient estimates and calibration results were perfect (Table 1, Figure 1E). · plain
“Calibration results for Examples 1 and 3 are unstable, in particular the flexible calibration curve.”
Quote in the paperSubstantial claim - Also stated in: § 2.2. Illustration: Examples 1-5 For illustration, we consider five simulated examples, as previously presented [5].We randomly generate four independent predictor variables to .These predictor variables are ordinal with three categories (-1, 0, and 1) that each have 33% prevalence in order to visualize calibration by covariate pattern.Let outcome be generated by an underlying logistic regression model with the true linear predictor , hence the intercept equals 0, and to are assumed to have linear main effects [5].In simulations, the true probability of event is calculated using the linear predictor and the observed outcome is generated as a Bernoulli variable from .Then, we develop a model to predict based on to , which means that we are using the correct model formulation.Finally, we validate on a new dataset that is generated with the exact same procedure.We repeat this process four times to illustrate the influence of random variability: (1) using 100 simulated patients for development and 100 for validation, (2) using 100 for development and 10,000 for validation, (3) using 10,000 for development and 100 for validation, (4) using 10,000 for development and 10,000 for validation, and (5) using 10 million for development and 10 million for validation.Estimation of model coefficients was unstable when the development data contains only 100 patients (Table 1), despite having more than 10 events per variable (4 parameters, 44 non-events, 56 events).Risk estimates were overfitted, as evidenced by the calibration slope and calibration curves at validation (Example 2 in Table 1, and Figure 1B) as well as the calibration curves.Further, the calibration intercept was negative which is suggestive of general overestimation.Enlarging the development dataset alleviated the problems (Example 4 in Table 1, Figure 1D).Similar issues emerge when using a small validation dataset (Figures 1A and 1C.Calibration results for Examples 1 and 3 are unstable, in particular the flexible calibration curve.With 10 million patients the coefficient estimates and calibration results were perfect (Table 1, Figure 1E). · plain
- Findingown data
With a small development sample of 100 patients, model coefficient estimation was unstable despite having more than 10 events per variable.
“Estimation of model coefficients was unstable when the development data contains only 100 patients (Table 1), despite having more than 10 events per variable (4 parameters, 44 non-events, 56 events).”
Quote in the paperSubstantial claim - Findingown data
With a small development sample, risk estimates were overfitted, as evidenced by the calibration slope and calibration curves at validation.
“Risk estimates were overfitted, as evidenced by the calibration slope and calibration curves at validation (Example 2 in Table 1, and Figure 1B) as well as the calibration curves.”
Quote in the paperSubstantial claim - Premisecitation
Weak calibration may still result in harmful decision-making.
“it is an important observation that calibration in a weak sense may still result in harmful decisionmaking [18].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [18] Van Calster, 2015, Calibration of risk prediction models: impact on decision-analytic performance, Med Decis Making source
Not checked in the free run.
- Interpretationcitation
Moderate calibration is a realistic goal in epidemiologic research.
“First, it is a realistic goal in epidemiologic research, where empirical data sets are often of relatively limited size, and the signal to noise ratio is unfavorable [31].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [31] Ennis, 1998, A comparison of statistical learning methods on the Gusto database, Stat Med source
Not checked in the free run.
- Interpretationcitation
Moderate calibration does not imply that the prediction model is valid in a strong sense.
“In line with Vach's work [5], we find that moderate calibration does not imply that the prediction model is 'valid' in a strong sense.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [5] Vach, 2013, Calibration of clinical prediction rules does not just assess bias, J Clin Epidemiol source
Not checked in the free run.
- Interpretationreasoning
A true model can only be identified in an infinitely large dataset.
“Such a true model can only be identified in an infinitely large dataset: in utopia.”
Quote in the paperSubstantial claim - Premisecitation
Previous work has shown that miscalibration decreases clinical utility compared to moderate calibration.
“Previous work has shown that miscalibration decreases clinical utility compared to moderate calibration [18].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [18] Van Calster, 2015, Calibration of risk prediction models: impact on decision-analytic performance, Med Decis Making source
Not checked in the free run.
- Interpretationreasoning
Strong calibration is unrealistic in empirical medical research.
“We conclude that strong calibration, although desirable for individual risk communication, is unrealistic in empirical medical research.”
- Also stated in: § Abstract · plain
“We argue that this is unrealistic: the model type may be incorrect, at model development the linear predictor is only asymptotically unbiased, and all nonlinear and interaction effects should be correctly modeled.”
Quote in the paperSubstantial claim - Also stated in: § Abstract · plain
- Interpretationreasoning
If a model is moderately but not strongly calibrated, it may provide biased risk estimates depending on an individual patient's covariate values.
“If a model is moderately but not strongly calibrated, we may provide biased risk estimates depending on an individual patient's covariate values.”
Quote in the paperSubstantial claim - Premisecitation
Measurement error may not be transportable across various settings.
“Moreover, measurement error may not be transportable across various settings [29].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [29] Carroll, 2009, Nonparametric prediction in measurement error models, J Am Stat Assoc source
Not checked in the free run.
- Interpretationreasoning
Moderate calibration allows communication of a reliable average risk for patients with the same estimated risk.
“In contrast, a moderately calibrated model allows the communication of a reliable average risk for patients with the same estimated risk: among patients with an estimated risk of 70% on average 70 out of 100 have the event, although there may exist relevant subgroups with different covariate patterns and different event rates.”
Quote in the paperSubstantial claim - Interpretationnone given
Weak calibration is by definition achieved on the dataset on which the prediction model was developed if standard estimation methods are used, such as maximum likelihood for logistic regression models.
“this level of calibration is by definition achieved on the dataset on which the prediction model was developed if standard estimation methods are used, such as maximum likelihood for logistic regression models.”
Quote in the paperSubstantial claim - Interpretationreasoning
In internal validation, the calibration intercept is not relevant because the model is validated for the same setting, where the mean of predictions matches the mean event rate according to standard statistical estimation methods such as maximum likelihood.
“The calibration intercept is not relevant because internal validation implies that the model is validated for the same setting, where the mean of predictions matches the mean event rate according to standard statistical estimation methods such as maximum likelihood.”
Quote in the paperSubstantial claim - Premisecitation
Using maximum likelihood estimation yields only asymptotically unbiased estimates of individual coefficients.
“using maximum likelihood estimation, as is done for logistic regression and Cox proportional hazards regression for time to event outcomes, yields only asymptotically unbiased estimates of individual coefficients [23].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [23] Nemes, 2009, Bias in odds ratios by logistic regression modelling and sample size, BMC Med Res Methodol source
Not checked in the free run.
- Interpretationreasoning
Moderate calibration guarantees non-harmful decisions in terms of Net Benefit and decision curve analysis.
“Moderate calibration guaranties non-harmful decisions in terms of Net Benefit and decision curve analysis: proof”
Quote in the paperNot a substantial claim - Interpretationreasoning
With continuous predictors, finding the correct model is unrealistic.
“but the use of continuous predictors makes finding the correct model unrealistic.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Categorizing patients based on predictor values to assess calibration beyond moderate leads to small subgroups and unreliable results.
“This is an insightful exercise, but the 'curse of dimensionality' is still looming: increasingly detailed categorizations will inevitably lead to small subgroups and hence unreliable results.”
Quote in the paperSubstantial claim - Interpretationreasoning
Calibration-in-the-large is insufficient as the sole criterion.
“Calibration-in-the-large is insufficient as the sole criterion.”
Quote in the paperSubstantial claim - Interpretationreasoning
Assessing strong calibration by calculating the observed event rate for every covariate pattern is hardly ever feasible because of limited sample size and/or continuous predictors.
“In practice this approach is hardly ever feasible because of limited sample size and/or the presence of continuous predictors: in such situations there may be as many patients as there are distinct covariate patterns.”
Quote in the paperSubstantial claim
- May combine two claims. 14
- Classified as probably not a substantial claim. 9
- The quote only partly states this claim. 1
May combine two claims.
Findingown dataThe Example 5 model exhibits perfect strong calibration: both calibration curves and all covariate patterns coincide with the diagonal.
“This model exhibits perfect strong calibration: both calibration curves and all covariate patterns coincide with the diagonal.”
Quote in the paperNot a substantial claimMay combine two claims.
FindingreasoningIn Example 6, the model is weakly but not moderately calibrated on the development data.
“Thus, the model is weakly but not moderately calibrated on the development data.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe Example 8 model exhibits perfect moderate calibration but lacks strong calibration because results per covariate pattern are scattered around the diagonal.
“This model exhibits perfect moderate calibration but lacks strong calibration because results per covariate pattern are scattered around the diagonal.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataWith 10 million patients, coefficient estimates and calibration results were perfect.
“With 10 million patients the coefficient estimates and calibration results were perfect (Table 1, Figure 1E).”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Interpretationnone givenStrong calibration requires that the model form is correct.
“First, strong calibration requires that the model form (e.g. a generalized linear model such as logistic regression) is correct.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningRefitting should lead to moderate calibration, although this may also require reassessment of nonlinear effects.
“Refitting should lead to moderate calibration, although this may also require reassessment of nonlinear effects.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationIn reality, the true model is unknown, may not have a generalized linear form, or may not even exist.
“in reality, the true model is unknown, may not have a generalized linear form, or may not even exist [22].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [22] Chatfield, 1995, Model uncertainty, data mining and statistical inference, J R Stat Soc Ser A source
Not checked in the free run.
May combine two claims.
InterpretationreasoningAiming for strong calibration requires the assumption that the model formulation is fully correct, and that unbiased model coefficients as well as unbiased linear predictors are obtained.
“In sum, aiming for strong calibration requires the assumption that the model formulation is fully correct, and that unbiased model coefficients as well as unbiased linear predictors are obtained.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningLogistic recalibration addresses lack of weak calibration but does not guarantee moderate calibration unless all coefficients were biased by the same rate.
“This method addresses lack of weak calibration, but does not guarantee moderate calibration unless all coefficients were biased by the same rate.”
Quote in the paperSubstantial claimMay combine two claims. The quote only partly states this claim.
InterpretationreasoningModerate calibration guarantees that clinical decisions based on the model are not harmful, with Net Benefit not lower than default strategies.
“we prove (see Appendix) that clinical utility in terms of Net Benefit will not be lower than the default strategies that either classify everyone as having the event or as not having the event.”
- Also stated in: § 6. Moderate calibration: a pragmatic guarantee for non-harmful decision-making · plain
“moderate calibration guarantees that decision-making based on the model is not clinically harmful.”
Quote in the paperSubstantial claim- Also stated in: § 6. Moderate calibration: a pragmatic guarantee for non-harmful decision-making · plain
May combine two claims.
PremisecitationStrong interactions or nonlinearities may lead to miscalibration in the development sample although weak calibration is perfect.
“For example, strong interactions or nonlinearities may lead to miscalibration in the development sample although weak calibration is perfect [11].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [11] Austin, 2014, Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers, Stat Med source
Not checked in the free run.
Classified as probably not a substantial claim.
Interpretationnone givenStrong calibration requires fully correct modeling of all nonlinear and non-additive effects.
“Third, strong calibration not only requires correctly estimated regression coefficients for the main effects of model predictors, but also requires fully correct modeling of all nonlinear and non-additive effects.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone givenFor a moderately calibrated model, among patients with an estimated risk of r%, on average r out of 100 of these patients have the event.
“For a moderately calibrated model the following property holds, among patients with an estimated risk of % on average out of 100 of these patients have the event.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationThe odds of the risk threshold is the harm-to-benefit ratio, i.e. the ratio of the harm of a false positive and the benefit of a true positive classification.
“The odds of the risk threshold is the harm-to-benefit ratio, i.e. the ratio of the harm of a false positive and the benefit of a true positive classification[21].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [21] Pauker, 1975, Therapeutic decision making: a cost-benefit analysis, N Engl J Med source
Not checked in the free run.
May combine two claims.
InterpretationcitationModerate calibration prevents the model from becoming harmful, but strong calibration may further enhance clinical utility.
“This suggests that moderate calibration prevents the model from becoming harmful [18] but that strong calibration may further enhance the model's clinical utility.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [18] Van Calster, 2015, Calibration of risk prediction models: impact on decision-analytic performance, Med Decis Making source
Not checked in the free run.
May combine two claims.
InterpretationreasoningIntercept adjustment addresses calibration-in-the-large but does not guarantee weak calibration.
“This will only address calibration-in-the-large, but does not guarantee weak calibration.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
PremisecitationSimple models that are moderately but not strongly calibrated often have practical advantages such as transparency or ease of use.
“While still providing sensible risk predictions, simple models that are for example moderately but not strongly calibrated often have many practical advantages such as transparency or ease of use [33].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [33] Marewski, 2012, Heuristic decision making in medicine, Dialogues Clin Neurosci source
Not checked in the free run.
May combine two claims.
InterpretationreasoningAt least part of the measurement error is systematic, and further thwarts the concept of the existence of a true model.
“This means that at least part of the measurement error is systematic, and further thwarts the concept of the existence of a true model.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable).
PremisecitationCalibration is a pivotal aspect of model performance.
“Calibration is a pivotal aspect of model performance [2][3][4]”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [2] Steyerberg, 2009
Not checked in the free run.
- cites [3] Kim, 2011, Probabilistic classifiers with high-dimensional data, Biostatistics source
Not checked in the free run.
- cites [4] Pepe, 2013, Methods for evaluating prediction performance of biomarkers and tests
Not checked in the free run.
May combine two claims.
PremisecitationMeasurement error of predictors is common and can bias regression coefficients.
“measurement error of the predictors is in practice often ignored although it is known that this is a common phenomenon that can bias regression coefficients [28].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [28] Fuller, 1987
Not checked in the free run.
Classified as probably not a substantial claim.
InterpretationreasoningIf the model's Net Benefit is lower than that of a default strategy, then using the model to support decision-making can be considered as clinically harmful.
“If the model's Net Benefit is lower than that of a default strategy, then using the model to support decision-making can be considered as clinically harmful.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationIn previous simulations, the Hosmer-Lemeshow test showed such poor performance that it may not be recommended for routine use.
“In previous simulations, the Hosmer-Lemeshow test showed such poor performance that it may not be recommended for routine use [7,37].”
Quote in the paperNot a substantial claim0 of 2 sources checkedProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone givenNet Benefit accounts for the different consequences of true and false positives through the risk threshold, i.e. the risk at which one is indifferent about classifying a patient as having the event (and providing treatment) or not.
“Net Benefit accounts for the different consequences of true and false positives through the risk threshold, i.e. the risk at which one is indifferent about classifying a patient as having the event (and providing treatment) or not.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
Calibrated risk models are vital for valid decision support.
“Calibrated risk models are vital for valid decision support.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Calibration approaches can be classified into three increasingly stringent levels, labelled as mean, weak, and moderate calibration.
“These approaches can be classified into three increasingly stringent levels, labelled as mean, weak, and moderate calibration. 52”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [52]
All discussed calibration measures are semi-proper.
“All discussed calibration measures are semi-proper, 52”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [52]
Estimated probabilities with too much spread are on average too close to 0 and 1 (calibration slope < 1), whereas estimated probabilities with too little spread are on average too close to the prevalence (calibration slope > 1).
“Estimated probabilities with too much spread are on average too close to 0 and 1 (slope < 1), whereas estimated probabilities with too little spread are on average too close to the prevalence (slope > 1).”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [52]
Previous recommendations were to include at least 100 to 200 individuals in the smallest outcome category.
“Previous recommendations were to include at least 100 to 200 individuals in the smallest outcome category.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Figure 5 identifies the wrong example
MinorThe discussion says Figure 5 compares decision curves for Example 9, the miscalibrated model. The figure caption instead identifies Example 8, which the paper describes as moderately calibrated.
“Let us consider Example 9 (Figure 3C).The model presented is not calibrated in a weak sense.We derived decision curves to assess the clinical utility of (1) the original miscalibrated model, (2) a recalibrated model to ensure moderate calibration, and (3) the strongly calibrated true model (Figure 5).”
“Figure: 5 Figure 5 . Figure 5. Decision curves for Example 8 to assess clinical usefulness.”
Cites
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270 · see citing works
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Self-citationClinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Self-citationStefanski, L. A., Fuller, W. A., & Wiley, J. (1988) · Journal of Business and Economic Statistics · doi:10.2307/1391893 · Cited by 2,835 · see citing works
Dichotomizing continuous predictors in multiple regression: a bad idea
Royston, P., Altman, D. G., & Sauerbrei, W. (2006) · Statistics in Medicine · doi:10.1002/sim.2331 · Cited by 2,312 · see citing works
Rao, S. J. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s263 · Cited by 1,986 · see citing works
Model uncertainty, data mining and statistical inference
Chatfield, C. (1995) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2983440 · Cited by 1,103 · see citing works
The number of subjects per variable required in linear regression analyses
Austin, P. C., & Steyerberg, E. W. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.12.014 · Cited by 1,067 · see citing works
Self-citationClinical Prediction Models: A Practical Approach to Development, Validation and Updating
Mann, C. J. H. (2009) · Kybernetes · doi:10.1108/k.2009.06738fae.002 · Cited by 794 · see citing works
Collins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729 · see citing works
Collins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665 · see citing works
Vergouwe, Y., Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.06.017 · Cited by 636 · see citing works
Self-citationTwo further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556 · see citing works
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552 · see citing works
Self-citationTherapeutic decision making: a cost-benefit analysis
Pauker, S. G., & Kassirer, J. P. (1975) · New England Journal of Medicine · doi:10.1056/nejm197507312930505 · Cited by 507 · see citing works
Satorra, A., & Fuller, W. A. (1988) · Journal of Educational Statistics · doi:10.2307/1164709 · Cited by 467 · see citing works
Bias in odds ratios by logistic regression modelling and sample size
Nemes, S., Jonasson, J. M., Genell, A., & Steineck, G. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-56 · Cited by 437 · see citing works
Heuristic decision making in medicine
Marewski, J. N., & Gigerenzer, G. (2012) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2012.14.1/jmarewski · Cited by 377 · see citing works
Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371 · see citing works
Austin, P. C., & Steyerberg, E. W. (2014) · Statistics in Medicine · doi:10.1002/sim.5941 · Cited by 362 · see citing works
Self-citationAlonzo, T. A. (2009) · American Journal of Epidemiology · doi:10.1093/aje/kwp129 · Cited by 190 · see citing works
Goodness-of-fit processes for logistic regression: simulation results
Hosmer, D. W., & Hjort, N. L. (2002) · Statistics in Medicine · doi:10.1002/sim.1200 · Cited by 164 · see citing works
Validation of probabilistic predictions
Miller, M. E., Langefeld, C. D., Tierney, W. M., Hui, S. L., & McDonald, C. J. (1993) · Medical Decision Making · doi:10.1177/0272989x9301300107 · Cited by 129 · see citing works
Application of Shrinkage Techniques in Logistic Regression Analysis: A Case Study
Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2001) · Statistica Neerlandica · doi:10.1111/1467-9574.00157 · Cited by 124 · see citing works
Self-citationEverything you always wanted to know about evaluating prediction models (but were too afraid to ask)
Vickers, A. J., & Cronin, A. M. (2010) · Urology · doi:10.1016/j.urology.2010.06.019 · Cited by 112 · see citing works
An evaluation of penalised survival methods for developing prognostic models with rare events
Ambler, G., Seaman, S. R., & Omar, R. (2011) · Statistics in Medicine · doi:10.1002/sim.4371 · Cited by 108 · see citing works
Decarli, A., Calza, S., Masala, G., Specchia, C., Palli, D., & Gail, M. H. (2006) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djj463 · Cited by 93 · see citing works
Shrinkage and Penalized Likelihood as Methods to Improve Predictive Accuracy
van Houwelingen, H. C. (2001) · Statistica Neerlandica · doi:10.1111/1467-9574.00154 · Cited by 86 · see citing works
A spline-based tool to assess and visualize the calibration of multiclass risk predictions
Van Hoorde, K., Van Huffel, S., Timmerman, D., Bourne, T., & Van Calster, B. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.12.016 · Cited by 85 · see citing works
Evaluation of Markers and Risk Prediction Models
CALSTER, B. V., Vickers, A. J., Pencina, M. J., Baker, S. G., Timmerman, D., & Steyerberg, E. W. (2013) · Medical Decision Making · doi:10.1177/0272989x12470757 · Cited by 76 · see citing works
Self-citationAnkerst, D. P., Boeck, A., Freedland, S. J., Thompson, I. M., Cronin, A. M., Roobol, M. J., Hugosson, J., Jones, J. S., Kattan, M. W., Klein, E. A., et al. (2011) · World Journal of Urology · doi:10.1007/s00345-011-0818-5 · Cited by 65 · see citing works
Regression Models as a Tool in Medical Research
Vach, W. (2012) · Chapman & Hall/CRC eBooks · doi:10.1201/b12925 · Cited by 56 · see citing works
Nonparametric prediction in measurement error models
Carroll, R. J., Delaigle, A., & Hall, P. (2009) · Journal of the American Statistical Association · doi:10.1198/jasa.2009.tm07543 · Cited by 51 · see citing works
Methods for Evaluating Prediction Performance of Biomarkers and Tests
PEPE, M. S., & Janes, H. (2013) · Lecture notes in statistics · doi:10.1007/978-1-4614-8981-8_7 · Cited by 44 · see citing works
Risk Assessment and Evaluation of Predictions
Lee, M. T., Gail, M. H., Pfeiffer, R. M., Satten, G. A., Cai, T., & Gandy, A. (2013) · Lecture notes in statistics · doi:10.1007/978-1-4614-8981-8 · Cited by 40 · see citing works
Calibration of clinical prediction rules does not just assess bias
Vach, W. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.003 · Cited by 26 · see citing works
Probabilistic classifiers with high-dimensional data
Kim, K. I., & Simon, R. (2011) · Biostatistics · doi:10.1093/biostatistics/kxq069 · Cited by 23 · see citing works
Bootstrap confidence intervals for loess-based calibration curves
Austin, P. C., & Steyerberg, E. W. (2014) · Statistics in Medicine · doi:10.1002/sim.6167 · Cited by 19 · see citing works
Self-citationSimple dichotomous updating methods improved the validity of polytomous prediction models
Hoorde, K. V., Vergouwe, Y., Timmerman, D., Huffel, S. V., Steyerberg, E. W., & CALSTER, B. V. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.04.014 · Cited by 11 · see citing works
Self-citationHosmer (1997) · doi:10.1002/(sici)1097-0258(19970515)16:9
Ennis (1998) · doi:10.1002/(sici)1097-0258(19981115)17:21
Flagged first, then by citations
Cited by
Sorted by citations
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · see citing works
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · see citing works
Self-citationChristodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119 · see citing works
Self-citationMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669 · see citing works
Alba, A. C., Agoritsas, T., Walsh, M., Hanna, S., Iorio, A., Devereaux, P. J., McGinn, T., & Guyatt, G. (2017) · JAMA · doi:10.1001/jama.2017.12126 · Cited by 1,631 · see citing works
Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators
CALSTER, B. V., Wynants, L., Verbeek, J. F. M., Verbakel, J. Y. J., Christodoulou, E., Vickers, A. J., Roobol, M. J., & Steyerberg, E. W. (2018) · European Urology · doi:10.1016/j.eururo.2018.08.038 · Cited by 1,289 · see citing works
Self-citationRiley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585 · see citing works
Second opinion needed: Communicating uncertainty in medical machine learning
Kompa, B., Snoek, J., & Beam, A. L. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00367-3 · Cited by 514 · see citing works
A tutorial on calibration measurements and calibration models for clinical prediction models
Huang, Y., Li, W., Macheret, F., Gabriel, R. A., & Ohno-Machado, L. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz228 · Cited by 486 · see citing works
Evaluation of clinical prediction models (part 2): how to undertake an external validation study
Riley, R., Archer, L., Snell, K., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L., & Collins, G. S. (2024) · British medical journal · doi:10.1136/bmj-2023-074820 · Cited by 515 · see citing works
Minimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400 · see citing works
Gerry, S., Bonnici, T., Birks, J., Kirtley, S., Virdee, P. S., Watkinson, P., & Collins, P. G. S. (2020) · BMJ · doi:10.1136/bmj.m1501 · Cited by 398 · see citing works
van den Goorbergh, R., van Smeden, M., Timmerman, D., & CALSTER, B. V. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac093 · Cited by 375 · see citing works
Timmerman, D., CALSTER, B. V., Testa, A. C., Savelli, L., Fischerová, D., Froyman, W., Wynants, L., Holsbeke, C. V., Epstein, E., Franchi, D., et al. (2016) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2016.01.007 · Cited by 337 · see citing works
Calibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312 · see citing works
Austin, P. C., & Steyerberg, E. W. (2019) · Statistics in Medicine · doi:10.1002/sim.8281 · Cited by 311 · see citing works
Self-citationThere is no such thing as a validated prediction model
Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M. (2023) · BMC Medicine · doi:10.1186/s12916-023-02779-w · Cited by 286 · see citing works
Self-citationten Haaf, K., Jeon, J., Tammemägi, M. C., Han, S. S., Kong, C. Y., Plevritis, S. K., Feuer, E. J., de Koning, H. J., Steyerberg, E. W., & Meza, R. (2017) · PLoS Medicine · doi:10.1371/journal.pmed.1002277 · Cited by 282 · see citing works
Self-citationCabitza, F., Campagner, A., Soares, F., García de Guadiana-Romualdo, L., Challa, F., Sulejmani, A., Seghezzi, M., & Carobene, A. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106288 · Cited by 284 · see citing works
Cowley, L., Farewell, D. M., Maguire, S., & Kemp, A. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0060-y · Cited by 265 · see citing works
Nordström, T., Discacciati, A., Bergman, M., Clements, M., Aly, M., Annerstedt, M., Glaessgen, A., Carlsson, S., Jäderling, F., Eklund, M., et al. (2021) · The Lancet Oncology · doi:10.1016/s1470-2045(21)00348-x · Cited by 214 · see citing works
Stability of clinical prediction models developed using statistical or machine learning methods
Riley, R. D., & Collins, G. S. (2023) · Biometrical Journal · doi:10.1002/bimj.202200302 · Cited by 211 · see citing works
Predictive analytics in health care: how can we know it works?
Van Calster, B., Wynants, L., Timmerman, D., Steyerberg, E. W., & Collins, G. S. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz130 · Cited by 223 · see citing works
Self-citationRiley, R. D., Snell, K. I. E., Archer, L., Ensor, J., Debray, T. P. A., CALSTER, B. V., van Smeden, M., & Collins, P. G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074821 · Cited by 213 · see citing works
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198 · see citing works
Gravesteijn, B. Y., Nieboer, D., Ercole, A., Lingsma, H. F., Nelson, D., CALSTER, B. V., Steyerberg, E. W., Åkerlund, C. A. I., Amrein, K., Anđelić, N., et al. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.03.005 · Cited by 184 · see citing works
Self-citation10-year performance of four models of breast cancer risk: a validation study
Terry, M. B., Liao, Y., Whittemore, A. S., Leoce, N., Buchsbaum, R., Zeinomar, N., Dite, G. S., Chung, W. K., Knight, J. A., Southey, M. C., et al. (2019) · The Lancet Oncology · doi:10.1016/s1470-2045(18)30902-1 · Cited by 170 · see citing works
Poor performance of clinical prediction models: the harm of commonly applied methods
Steyerberg, E. W., Uno, H., Ioannidis, J. P. A., CALSTER, B. V., Ukaegbu, C. I., Dhingra, T. G., Syngal, S., & Kastrinos, F. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.013 · Cited by 166 · see citing works
Self-citationMcLernon, D. J., Giardiello, D., CALSTER, B. V., Wynants, L., van Geloven, N., van Smeden, M., Therneau, T. M., Steyerberg, E. W., Initiative, T. G. 6. A. 8. O. T. S., McLernon, D. J., et al. (2022) · Annals of Internal Medicine · doi:10.7326/m22-0844 · Cited by 167 · see citing works
Self-citationOakland, K., Jairath, V., Uberoi, R., Guy, R., Ayaru, L., Mortensen, N., Murphy, M. F., & Collins, P. G. S. (2017) · The Lancet. Gastroenterology & hepatology · doi:10.1016/s2468-1253(17)30150-4 · Cited by 164 · see citing works
Detection of calibration drift in clinical prediction models to inform model updating
Davis, S. E., Greevy, R. A., Lasko, T. A., Walsh, C. G., & Matheny, M. E. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103611 · Cited by 160 · see citing works
Joint EANM/SNMMI guideline on radiomics in nuclear medicine
Hatt, M., Krizsan, A. K., Rahmim, A., Bradshaw, T., Costa, P. F., Forgács, A., Seifert, R., Zwanenburg, A., Naqa, I. E., Kinahan, P. E., et al. (2022) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-022-06001-6 · Cited by 156 · see citing works
Validation of clinical prediction models: what does the “calibration slope” really measure?
Stevens, R. J., & Poppe, K. K. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.09.016 · Cited by 146 · see citing works
Riley, R. D., Snell, K. I. E., Martin, G. P., Whittle, R., Archer, L., Sperrin, M., & Collins, G. S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.12.005 · Cited by 142 · see citing works
Sufriyana, H., Husnayain, A., Chen, Y.-L., Kuo, C.-Y., Singh, O., Yeh, T.-Y., Wu, Y., & Su, E. C. (2020) · JMIR Medical Informatics · doi:10.2196/16503 · Cited by 145 · see citing works
Verbakel, J. Y., Steyerberg, E. W., Uno, H., De Cock, B., Wynants, L., Collins, G. S., & Van Calster, B. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.028 · Cited by 138 · see citing works
Self-citationPredicting risk of postpartum haemorrhage: a systematic review
Neary, C., Naheed, S., McLernon, D. J., & Black, M. E. (2020) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.16379 · Cited by 135 · see citing works
The current status of risk-stratified breast screening
Clift, A. K., Dodwell, D. J., Lord, S., Petrou, S., Brady, M., Collins, P. G. S., & Hippisley‐Cox, J. (2021) · British Journal of Cancer · doi:10.1038/s41416-021-01550-3 · Cited by 134 · see citing works
Snell, K. I. E., Ensor, J., Debray, T. P. A., Moons, K. G. M., & Riley, R. D. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280217705678 · Cited by 130 · see citing works
CALSTER, B. V., Valentin, L., Froyman, W., Landolfo, C., Ceusters, J., Testa, A. C., Wynants, L., Sladkevicius, P., Holsbeke, C. V., Domali, E., et al. (2020) · BMJ · doi:10.1136/bmj.m2614 · Cited by 128 · see citing works
Luijken, K., Groenwold, R., van Calster, B., Steyerberg, E., & van Smeden, M. (2019) · Statistics in Medicine · doi:10.1002/sim.8183 · Cited by 103 · see citing works
Self-citationLamain-de Ruiter, M., Kwee, A., Naaktgeboren, C. A., de Groot, I., Evers, I. M., Groenendaal, F., Hering, Y. R., Huisjes, A. J. M., Kirpestein, C., Monincx, W. M., et al. (2016) · BMJ · doi:10.1136/bmj.i4338 · Cited by 123 · see citing works
Feinstein, M. J., Nance, R. M., Drozd, D. R., Ning, H., Delaney, J. A., Heckbert, S. R., Budoff, M. J., Mathews, W. C., Kitahata, M. M., Saag, M. S., et al. (2016) · JAMA Cardiology · doi:10.1001/jamacardio.2016.4494 · Cited by 121 · see citing works
Snell, K. I. E., Archer, L., Ensor, J., Bonnett, L. J., Debray, T. P. A., Phillips, B., Collins, P. G. S., & Riley, R. D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.011 · Cited by 125 · see citing works
Van Calster, B., van Smeden, M., De Cock, B., & Steyerberg, E. W. (2020) · Statistical Methods in Medical Research · doi:10.1177/0962280220921415 · Cited by 120 · see citing works
Self-citationTargeted validation: validating clinical prediction models in their intended population and setting
Sperrin, M., Riley, R. D., Collins, P. G. S., & Martin, G. P. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00136-8 · Cited by 115 · see citing works
Rahmani, K., Thapa, R., Tsou, P., Chetty, S. C., Barnes, G., Lam, C., & Tso, C. F. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104930 · Cited by 115 · see citing works
Riley, R. D., Collins, P. G. S., Ensor, J., Archer, L., Booth, S., Mozumder, S. I., Rutherford, M. J., van Smeden, M., Lambert, P. C., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9275 · Cited by 116 · see citing works
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works
Self-citationKlaveren, D. V., Steyerberg, E. W., Serruys, P. W. J. C., & Kent, D. M. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.021 · Cited by 111 · see citing works
Self-citationA Prediction Model to Identify Febrile Infants ≤60 Days at Low Risk of Invasive Bacterial Infection
Aronson, P. L., Shabanova, V., Shapiro, E. D., Wang, M. E., Nigrovic, L. E., Pruitt, C. M., DePorre, A. G., Leazer, R. C., Desai, S., Sartori, L. F., et al. (2019) · PEDIATRICS · doi:10.1542/peds.2018-3604 · Cited by 105 · see citing works
Licher, S., Yilmaz, P., Leening, M. J. G., Wolters, F. J., Vernooij, M. W., Stephan, B. C. M., Ikram, M. K., & Ikram, M. A. (2018) · European Journal of Epidemiology · doi:10.1007/s10654-018-0403-y · Cited by 103 · see citing works
Sayasneh, A., Ferrara, L., Cock, B. D., Saso, S., Al‐Memar, M., Johnson, S., Kaijser, J., Carvalho, J., Husicka, R., Smith, A. A., et al. (2016) · British Journal of Cancer · doi:10.1038/bjc.2016.227 · Cited by 100 · see citing works
Which Models Can I Use to Predict Adult ICU Length of Stay? A Systematic Review*
Verburg, I. W. M., Atashi, A., Eslami, S., Holman, R., Abu–Hanna, A., de Jonge, E., Peek, N. B., & de Keizer, N. F. (2016) · Critical Care Medicine · doi:10.1097/ccm.0000000000002054 · Cited by 98 · see citing works
Clift, A. K., Dodwell, D., Lord, S., Petrou, S., Brady, M., Collins, G. S., & Hippisley-Cox, J. (2023) · BMJ · doi:10.1136/bmj-2022-073800 · Cited by 97 · see citing works
A systematic review of the quality of clinical prediction models in in vitro fertilisation
Ratna, M. B., Bhattacharya, S., Abdulrahim, B., & McLernon, D. J. (2019) · Human Reproduction · doi:10.1093/humrep/dez258 · Cited by 94 · see citing works
Key steps and common pitfalls in developing and validating risk models
Wynants, L., Collins, P. G. S., & CALSTER, B. V. (2016) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.14170 · Cited by 95 · see citing works
Prognostic models will be victims of their own success, unless…
Lenert, M. C., Matheny, M. E., & Walsh, C. G. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz145 · Cited by 90 · see citing works
Karhade, A., Ahmed, A., Pennington, Z., Chara, A., Schilling, A. T., Thio, Q., Ogink, P., Sciubba, D., & Schwab, J. (2019) · The spine journal · doi:10.1016/j.spinee.2019.09.003 · Cited by 84 · see citing works
Debray, T. P. A., Collins, P. G. S., Riley, R. D., Snell, K. I. E., CALSTER, B. V., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071018 · Cited by 97 · see citing works
Yang, C. B., Fridgeirsson, E. A., Kors, J. A., Reps, J., & Rijnbeek, P. R. (2024) · Journal Of Big Data · doi:10.1186/s40537-023-00857-7 · Cited by 91 · see citing works
Mosquera, C., Ferrer, L., Milone, D. H., Luna, D., & Ferrante, E. (2024) · European Radiology · doi:10.1007/s00330-024-10834-0 · Cited by 92 · see citing works
Lee, A. S. Y., Mu, J., Joynt, G. M., Chiu, C. H., Lai, V. K. W., Gin, T., & Underwood, M. J. (2017) · British Journal of Anaesthesia · doi:10.1093/bja/aew476 · Cited by 85 · see citing works
Ensor, J., Riley, R. D., Moore, D., Snell, K. I. E., Bayliss, S., & Fitzmaurice, D. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011190 · Cited by 84 · see citing works
Systematic review of prediction models for pulmonary tuberculosis treatment outcomes in adults
Peetluk, L. S., Ridolfi, F. M., Rebeiro, P. F., Liu, D., Rolla, V. C., & Sterling, T. R. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044687 · Cited by 83 · see citing works
Wessler, B. S., Nelson, J., Park, J. G., McGinnes, H., Gulati, G., Brazil, R., Van Calster, B., van Klaveren, D., Venema, E., Steyerberg, E., et al. (2021) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.007858 · Cited by 83 · see citing works
Self-citationIntroduction to Artificial Intelligence and Machine Learning for Pathology
Harrison, J. H., Gilbertson, J. R., Hanna, M. G., Olson, N. H., Seheult, J. N., Sorace, J. M., & Stram, M. N. (2021) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2020-0541-cp · Cited by 83 · see citing works
Oosterhoff, J. H. F., & Doornberg, J. N. (2020) · EFORT Open Reviews · doi:10.1302/2058-5241.5.190092 · Cited by 81 · see citing works
Personalising lung cancer screening: An overview of risk‐stratification opportunities and challenges
ten Haaf, K., van der Aalst, C. M., de Koning, H. J., Kaaks, R., & Tammemägi, M. C. (2021) · International Journal of Cancer · doi:10.1002/ijc.33578 · Cited by 80 · see citing works
A Proteomics-Based Approach for Prediction of Different Cardiovascular Diseases and Dementia
Ho, F. K. W., Mark, P. B., Lees, J. S., Pell, J. P., Strawbridge, R. J., Kimenai, D. M., Mills, N. L., Woodward, M., McMurray, J. J. V., Sattar, N. A., et al. (2024) · Circulation · doi:10.1161/circulationaha.124.070454 · Cited by 82 · see citing works
Haines, K. J., Hibbert, E., McPeake, J. M., Anderson, B. J., Bienvenu, O. J., Andrews, A., Brummel, N. E., Ferrante, L. E., Hopkins, R. O., Hough, C. L., et al. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004659 · Cited by 75 · see citing works
Peeken, J. C., Asadpour, R., Specht, K., Chen, E. Y., Klymenko, O., Akinkuoroye, V., Hippe, D. S., Spraker, M. B., Schaub, S. K., Dapper, H., et al. (2021) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2021.08.023 · Cited by 74 · see citing works
Steyerberg, E. W., Nieboer, D., Debray, T. P. A., & van Houwelingen, H. C. (2019) · Statistics in Medicine · doi:10.1002/sim.8296 · Cited by 73 · see citing works
Self-citationCalibrating machine learning approaches for probability estimation: A comprehensive comparison
Ojeda, F. M., Jansen, M. L., Thiéry, A., Blankenberg, S., Weimar, C., Schmid, M., & Ziegler, A. (2023) · Statistics in Medicine · doi:10.1002/sim.9921 · Cited by 71 · see citing works
Mortality prediction models in the adult critically ill: A scoping review
Keuning, B. E., Kaufmann, T., Wiersema, R., Granholm, A., Pettilä, V., Møller, M. H., Christiansen, C. F., Forte, J. C., Snieder, H., Keus, F., et al. (2019) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.13527 · Cited by 71 · see citing works
Mikolić, A., Polinder, S., Steyerberg, E. W., Helmrich, I. R., Giacino, J. T., Maas, A. I. R., van der Naalt, J., Voormolen, D. C., von Steinbüchel, N., Wilson, L., et al. (2020) · Journal of Neurotrauma · doi:10.1089/neu.2020.7074 · Cited by 70 · see citing works
Self-citationInternational multicenter validation of AI-driven ultrasound detection of ovarian cancer
Christiansen, F., Konuk, E., Ganeshan, A. R., Welch, R., Huix, J. P., Czekierdowski, A., Leone, F. P. G., Haak, L. A., Fruscio, R., Gaurilčikas, A., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03329-4 · Cited by 72 · see citing works
Wijnhoven, H. A. H., Elstgeest, L. E. M., de Vet, H. C. W., Nicolaou, M., Snijder, M. B., & Visser, M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0196406 · Cited by 66 · see citing works
A Validated Nomogram to Predict Upstaging of Ductal Carcinoma in Situ to Invasive Disease
Jakub, J. W., Murphy, B. L., Gonzalez, A. B., Conners, A. L., Henrichsen, T. L., Maimone, S., Keeney, M. G., McLaughlin, S. A., Pockaj, B. A., Chen, B., et al. (2017) · Annals of Surgical Oncology · doi:10.1245/s10434-017-5927-y · Cited by 65 · see citing works
Granholm, A., Perner, A., Krag, M., Hjortrup, P. B., Haase, N., Holst, L. B., Marker, S., Collet, M. O., Jensen, A. K. G., & Møller, M. H. (2017) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.13048 · Cited by 64 · see citing works
Trier, T. J. V., Snaterse, M., Boekholdt, S. M., Reimer, W. J. M. S. O., Hageman, S. H. J., Visseren, F. L. J., Dorresteijn, J. A. N., Peters, R. J. G., & Jorstad, H. T. (2023) · European Journal of Preventive Cardiology · doi:10.1093/eurjpc/zwad318 · Cited by 64 · see citing works
Rothenberg, K. A., George, E., Trickey, A. W., Barreto, N. B., Johnson, T. M., Hall, D. E., Johanning, J. M., & Arya, S. (2020) · Annals of Vascular Surgery · doi:10.1016/j.avsg.2020.01.015 · Cited by 63 · see citing works
Challenges of developing a cardiovascular risk calculator for patients with rheumatoid arthritis
Crowson, C. S., Rollefstad, S., Kitas, G. D., van Riel, P. L. C. M., Gabriel, S. E., & Semb, A. G. (2017) · PLoS ONE · doi:10.1371/journal.pone.0174656 · Cited by 63 · see citing works
Fronczek, J., Polok, K., Devereaux, P. J., Górka, J., Archbold, A., Biccard, B. M., Duceppe, E., Manach, Y. L., Sessler, D. I., Duchińska, M., et al. (2019) · British Journal of Anaesthesia · doi:10.1016/j.bja.2019.05.029 · Cited by 62 · see citing works
Clinical prediction models and the multiverse of madness
Riley, R. D., Pate, A., Dhiman, P., Archer, L., Martin, G. P., & Collins, P. G. S. (2023) · BMC Medicine · doi:10.1186/s12916-023-03212-y · Cited by 63 · see citing works
Kremers, F. C. C., Venema, E., Duvekot, M. H. C., Yo, L., Bokkers, R. P. H., Nijeholt, G. J. L. À., van Es, A., van der Lugt, A., Majoie, C. B. L. M., Burke, J., et al. (2021) · Stroke · doi:10.1161/strokeaha.120.033445 · Cited by 60 · see citing works
Vizzutti, F., Celsa, C., Calvaruso, V., Enea, M., Battaglia, S. L., Turco, L., Senzolo, M., Nardelli, S., Miraglia, R., Roccarina, D., et al. (2022) · Hepatology · doi:10.1002/hep.32704 · Cited by 60 · see citing works
Debray, T. P. A., Collins, P. G. S., Riley, R. D., Snell, K. I. E., CALSTER, B. V., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071058 · Cited by 62 · see citing works
Genders, T. S. S., Coles, A., Hoffmann, U., Patel, M. R., Mark, D. B., Lee, K. L., Steyerberg, E. W., Hunink, M. G. M., & Douglas, P. S. (2017) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2017.02.020 · Cited by 59 · see citing works
Self-citationThe number needed to benefit: estimating the value of predictive analytics in healthcare
Liu, V. X., Bates, D. W., Wiens, J., & Shah, N. H. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz088 · Cited by 59 · see citing works
Bongers, M., Karhade, A., Villavieja, J., Groot, O., Bilsky, M., Laufer, I., & Schwab, J. (2020) · The spine journal · doi:10.1016/j.spinee.2020.05.003 · Cited by 51 · see citing works
Gneiting, T., & Resin, J. (2023) · Electronic Journal of Statistics · doi:10.1214/23-ejs2180 · Cited by 58 · see citing works
Dijkland, S. A., Helmrich, I. R., Nieboer, D., van der Jagt, M., Dippel, D. W. J., Menon, D. K., Stocchetti, N., Maas, A. I. R., Lingsma, H. F., Steyerberg, E. W., et al. (2020) · Journal of Neurotrauma · doi:10.1089/neu.2020.7300 · Cited by 56 · see citing works
Self-citationYang, J.-J., Chen, C.-W., Fourman, M. S., Bongers, M., Karhade, A., Groot, O., Lin, W.-H., Yen, H.-K., Huang, P.-H., Yang, S.-H., et al. (2021) · The spine journal · doi:10.1016/j.spinee.2021.01.027 · Cited by 49 · see citing works
Afshar, M., Sharma, B., Dligach, D., Oguss, M., Brown, R., Chhabra, N., Thompson, H. M., Markossian, T., Joyce, C., Churpek, M. M., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00041-3 · Cited by 55 · see citing works
Cheng, Y., Gadd, D. A., Gieger, C., Monterrubio‐Gómez, K., Zhang, Y., Berta, I., Stam, M. J., Szlachetka, N., Lobzaev, E., Wrobel, N., et al. (2023) · Nature Aging · doi:10.1038/s43587-023-00391-4 · Cited by 55 · see citing works
Nightingale Health Biobank Collaborative Group, Barrett, J. C., Esko, T., Fischer, K., Jostins-Dean, L., Jousilahti, P., Julkunen, H., Jääskeläinen, T., Kangas, A. J., Kerimov, N., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-54357-0 · Cited by 72 · see citing works
Tufts PACE Clinical Predictive Model Registry: update 1990 through 2015
Wessler, B. S., Paulus, J. K., Lundquist, C. M., Ajlan, M., Natto, Z. S., Janes, W., Jethmalani, N., Raman, G., Lutz, J. S., & Kent, D. M. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-017-0021-2 · Cited by 57 · see citing works
Modified Brier score for evaluating prediction accuracy for binary outcomes
Yang, W., Jiang, J.-K., Schnellinger, E., Kimmel, S., & Guo, W. (2022) · Statistical Methods in Medical Research · doi:10.1177/09622802221122391 · Cited by 61 · see citing works
Uncertainty of risk estimates from clinical prediction models: rationale, challenges, and approaches
Riley, R. D., Collins, P. G. S., Kirton, L., Snell, K. I. E., Ensor, J., Whittle, R., Dhiman, P., van Smeden, M., Liu, X., Alderman, J. E., et al. (2025) · BMJ · doi:10.1136/bmj-2024-080749 · Cited by 54 · see citing works
- Authors
- Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W.
- Affiliations
- Erasmus MC (NL)
- Duke University (US)
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2016-06
- Volume
- 74
- Pages
- 167-176
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2225109326
- Cited by
- 910
- References
- 42
- Citations by year
- 2026: 192 · 2025: 99 · 2024: 118 · 2023: 93 · 2022: 125 · 2021: 91 · 2020: 58 · 2019: 49 · 2018: 35 · 2017: 30 · 2016: 19 · 2013: 1
- FWCI
- 44.92
- Type
- journal article
- Language
- en
- Open access
- green
- lirias.kuleuven.be
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Lirias (KU Leuven) (submittedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Funders
- Fonds Wetenschappelijk Onderzoek (G049312N)
- KU Leuven (C24/15/037)
- Vlaamse regering (G049312N, C24/15/037)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- risk model calibration, external validation, decision support, overfitting, simulation study
- MeSH terms
- Calibration, Computer Simulation, Decision Support Techniques, Humans, Risk, Reproducibility of Results, Models, Statistical, Bias, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
