The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index; none of the 17 sources its claims cite is checked for support yet.
A calibration hierarchy for risk models was defined: from utopia to empirical data
Open accessAbstract
Objective Calibrated risk models are vital for valid decision support. We define four levels of calibration and describe implications for model development and external validation of predictions.Study design and setting We present results based on simulated data sets.Results A common definition of calibration is "having an event rate of R% among patients with a predicted risk of R%," which we refer to as "moderate calibration." Weaker forms of calibration only require the average predicted risk (mean calibration) or the average prediction effects (weak calibration) to be correct. "Strong calibration" requires that the event rate equals the predicted risk for every covariate pattern. This implies that the model is fully correct for the validation setting. We argue that this is unrealistic: the model type may be incorrect, the linear predictor is only asymptotically unbiased, and all nonlinear and interaction effects should be correctly modeled. In addition, we prove that moderate calibration guarantees nonharmful decision making. Finally, results indicate that a flexible assessment of calibration in small validation data sets is problematic.Conclusion Strong calibration is desirable for individualized decision support but unrealistic and counter productive by stimulating the development of overly complex models. Model development and external validation should focus on moderate calibration.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
20 Own interpretation · 14 Not checked yet · 13 Own result · 5 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There was some miscalibration in mean risk for Examples 1-2 where the development sample was small, even though the correct underlying model formulation was estimated.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 3.1. Level 1: mean calibration (Calibration-in-the-large)
plaincorrelationaln = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Enlarging the development dataset alleviated the problems (Example 4 in Table 1, Figure 1D).”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 2.2. Illustration: Examples 1-5 For illustration, we consider five simulated examples, as previously presented [5].We randomly generate four independent predictor variables to .These predictor variables are ordinal with three categories (-1, 0, and 1) that each have 33% prevalence in order to visualize calibration by covariate pattern.Let outcome be generated by an underlying logistic regression model with the true linear predictor , hence the intercept equals 0, and to are assumed to have linear main effects [5].In simulations, the true probability of event is calculated using the linear predictor and the observed outcome is generated as a Bernoulli variable from .Then, we develop a model to predict based on to , which means that we are using the correct model formulation.Finally, we validate on a new dataset that is generated with the exact same procedure.We repeat this process four times to illustrate the influence of random variability: (1) using 100 simulated patients for development and 100 for validation, (2) using 100 for development and 10,000 for validation, (3) using 10,000 for development and 100 for validation, (4) using 10,000 for development and 10,000 for validation, and (5) using 10 million for development and 10 million for validation.Estimation of model coefficients was unstable when the development data contains only 100 patients (Table 1), despite having more than 10 events per variable (4 parameters, 44 non-events, 56 events).Risk estimates were overfitted, as evidenced by the calibration slope and calibration curves at validation (Example 2 in Table 1, and Figure 1B) as well as the calibration curves.Further, the calibration intercept was negative which is suggestive of general overestimation.Enlarging the development dataset alleviated the problems (Example 4 in Table 1, Figure 1D).Similar issues emerge when using a small validation dataset (Figures 1A and 1C.Calibration results for Examples 1 and 3 are unstable, in particular the flexible calibration curve.With 10 million patients the coefficient estimates and calibration results were perfect (Table 1, Figure 1E).
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results suggest that the flexible calibration curve is more variable and hence requires more data than the calibration slope for a stable assessment.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 8. Sample size at external validation
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is clear that observed calibration curves will easily deviate from the diagonal line even when the model is moderately or strongly calibrated.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 8. Sample size at external validation
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nevertheless, the decision curve for the true model is the best one.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 4. Calibration, decision-making, and clinical utility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Again, only the flexible calibration curve reveals that calibration is problematic.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 3.4. Missing non-linearity or interaction: examples 6-7
plaindirectn = 10,000- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The moderately recalibrated version does not have that problem anymore: Net Benefit is now at least as high as that of the default strategies.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 4. Calibration, decision-making, and clinical utility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The original model has Net Benefit below that of treat all or treat none for a subset of risk thresholds (Figure 5).”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 4. Calibration, decision-making, and clinical utility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For larger development datasets the disagreement between observed and predicted risks disappeared (Examples 3-5).”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 3.1. Level 1: mean calibration (Calibration-in-the-large)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“results indicate that a flexible assessment of calibration in small validation datasets is problematic.”
- Also stated in: § 2.2. Illustration: Examples 1-5 For illustration, we consider five simulated examples, as previously presented [5].We randomly generate four independent predictor variables to .These predictor variables are ordinal with three categories (-1, 0, and 1) that each have 33% prevalence in order to visualize calibration by covariate pattern.Let outcome be generated by an underlying logistic regression model with the true linear predictor , hence the intercept equals 0, and to are assumed to have linear main effects [5].In simulations, the true probability of event is calculated using the linear predictor and the observed outcome is generated as a Bernoulli variable from .Then, we develop a model to predict based on to , which means that we are using the correct model formulation.Finally, we validate on a new dataset that is generated with the exact same procedure.We repeat this process four times to illustrate the influence of random variability: (1) using 100 simulated patients for development and 100 for validation, (2) using 100 for development and 10,000 for validation, (3) using 10,000 for development and 100 for validation, (4) using 10,000 for development and 10,000 for validation, and (5) using 10 million for development and 10 million for validation.Estimation of model coefficients was unstable when the development data contains only 100 patients (Table 1), despite having more than 10 events per variable (4 parameters, 44 non-events, 56 events).Risk estimates were overfitted, as evidenced by the calibration slope and calibration curves at validation (Example 2 in Table 1, and Figure 1B) as well as the calibration curves.Further, the calibration intercept was negative which is suggestive of general overestimation.Enlarging the development dataset alleviated the problems (Example 4 in Table 1, Figure 1D).Similar issues emerge when using a small validation dataset (Figures 1A and 1C.Calibration results for Examples 1 and 3 are unstable, in particular the flexible calibration curve.With 10 million patients the coefficient estimates and calibration results were perfect (Table 1, Figure 1E). · plain
“Calibration results for Examples 1 and 3 are unstable, in particular the flexible calibration curve.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > Abstract
plainn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.2. Illustration: Examples 1-5 For illustration, we consider five simulated examples, as previously presented [5].We randomly generate four independent predictor variables to .These predictor variables are ordinal with three categories (-1, 0, and 1) that each have 33% prevalence in order to visualize calibration by covariate pattern.Let outcome be generated by an underlying logistic regression model with the true linear predictor , hence the intercept equals 0, and to are assumed to have linear main effects [5].In simulations, the true probability of event is calculated using the linear predictor and the observed outcome is generated as a Bernoulli variable from .Then, we develop a model to predict based on to , which means that we are using the correct model formulation.Finally, we validate on a new dataset that is generated with the exact same procedure.We repeat this process four times to illustrate the influence of random variability: (1) using 100 simulated patients for development and 100 for validation, (2) using 100 for development and 10,000 for validation, (3) using 10,000 for development and 100 for validation, (4) using 10,000 for development and 10,000 for validation, and (5) using 10 million for development and 10 million for validation.Estimation of model coefficients was unstable when the development data contains only 100 patients (Table 1), despite having more than 10 events per variable (4 parameters, 44 non-events, 56 events).Risk estimates were overfitted, as evidenced by the calibration slope and calibration curves at validation (Example 2 in Table 1, and Figure 1B) as well as the calibration curves.Further, the calibration intercept was negative which is suggestive of general overestimation.Enlarging the development dataset alleviated the problems (Example 4 in Table 1, Figure 1D).Similar issues emerge when using a small validation dataset (Figures 1A and 1C.Calibration results for Examples 1 and 3 are unstable, in particular the flexible calibration curve.With 10 million patients the coefficient estimates and calibration results were perfect (Table 1, Figure 1E). · plain
- It may state two separate claims as one. 14
- Probably not a substantial claim. 9
- The quote only partly states this claim. 1
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This model exhibits perfect strong calibration: both calibration curves and all covariate patterns coincide with the diagonal.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 3.6. Model misspecification: Examples 8-9
plainn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Thus, the model is weakly but not moderately calibrated on the development data.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 3.4. Missing non-linearity or interaction: examples 6-7
plainn = 10,000- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This model exhibits perfect moderate calibration but lacks strong calibration because results per covariate pattern are scattered around the diagonal.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 3.6. Model misspecification: Examples 8-9
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With 10 million patients the coefficient estimates and calibration results were perfect (Table 1, Figure 1E).”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 2.2. Illustration: Examples 1-5 For illustration, we consider five simulated examples, as previously presented [5].We randomly generate four independent predictor variables to .These predictor variables are ordinal with three categories (-1, 0, and 1) that each have 33% prevalence in order to visualize calibration by covariate pattern.Let outcome be generated by an underlying logistic regression model with the true linear predictor , hence the intercept equals 0, and to are assumed to have linear main effects [5].In simulations, the true probability of event is calculated using the linear predictor and the observed outcome is generated as a Bernoulli variable from .Then, we develop a model to predict based on to , which means that we are using the correct model formulation.Finally, we validate on a new dataset that is generated with the exact same procedure.We repeat this process four times to illustrate the influence of random variability: (1) using 100 simulated patients for development and 100 for validation, (2) using 100 for development and 10,000 for validation, (3) using 10,000 for development and 100 for validation, (4) using 10,000 for development and 10,000 for validation, and (5) using 10 million for development and 10 million for validation.Estimation of model coefficients was unstable when the development data contains only 100 patients (Table 1), despite having more than 10 events per variable (4 parameters, 44 non-events, 56 events).Risk estimates were overfitted, as evidenced by the calibration slope and calibration curves at validation (Example 2 in Table 1, and Figure 1B) as well as the calibration curves.Further, the calibration intercept was negative which is suggestive of general overestimation.Enlarging the development dataset alleviated the problems (Example 4 in Table 1, Figure 1D).Similar issues emerge when using a small validation dataset (Figures 1A and 1C.Calibration results for Examples 1 and 3 are unstable, in particular the flexible calibration curve.With 10 million patients the coefficient estimates and calibration results were perfect (Table 1, Figure 1E).
plainn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, strong calibration requires that the model form (e.g. a generalized linear model such as logistic regression) is correct.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 5. Strong calibration: realistic or utopic?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Refitting should lead to moderate calibration, although this may also require reassessment of nonlinear effects.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 7. A link with model updating
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“in reality, the true model is unknown, may not have a generalized linear form, or may not even exist [22].”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 5. Strong calibration: realistic or utopic?
What the sources say
- Not checked yetcites [22] Chatfield, 1995, Model uncertainty, data mining and statistical inference, J R Stat Soc Ser A source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In sum, aiming for strong calibration requires the assumption that the model formulation is fully correct, and that unbiased model coefficients as well as unbiased linear predictors are obtained.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 5. Strong calibration: realistic or utopic?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This method addresses lack of weak calibration, but does not guarantee moderate calibration unless all coefficients were biased by the same rate.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 7. A link with model updating
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we prove (see Appendix) that clinical utility in terms of Net Benefit will not be lower than the default strategies that either classify everyone as having the event or as not having the event.”
- Also stated in: § 6. Moderate calibration: a pragmatic guarantee for non-harmful decision-making · plain
“moderate calibration guarantees that decision-making based on the model is not clinically harmful.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 4. Calibration, decision-making, and clinical utility
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § 6. Moderate calibration: a pragmatic guarantee for non-harmful decision-making · plain
In the paper
“For example, strong interactions or nonlinearities may lead to miscalibration in the development sample although weak calibration is perfect [11].”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 3.3. Level 3: moderate calibration
What the sources say
- Not checked yetcites [11] Austin, 2014, Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers, Stat Med source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Third, strong calibration not only requires correctly estimated regression coefficients for the main effects of model predictors, but also requires fully correct modeling of all nonlinear and non-additive effects.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 5. Strong calibration: realistic or utopic?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For a moderately calibrated model the following property holds, among patients with an estimated risk of % on average out of 100 of these patients have the event.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > Additional information
plaincorrelationaln = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“The odds of the risk threshold is the harm-to-benefit ratio, i.e. the ratio of the harm of a false positive and the benefit of a true positive classification[21].”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > cex.d01
What the sources say
- Not checked yetcites [21] Pauker, 1975, Therapeutic decision making: a cost-benefit analysis, N Engl J Med source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“This suggests that moderate calibration prevents the model from becoming harmful [18] but that strong calibration may further enhance the model's clinical utility.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 4. Calibration, decision-making, and clinical utility
What the sources say
- Not checked yetcites [18] Van Calster, 2015, Calibration of risk prediction models: impact on decision-analytic performance, Med Decis Making source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This will only address calibration-in-the-large, but does not guarantee weak calibration.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 7. A link with model updating
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“While still providing sensible risk predictions, simple models that are for example moderately but not strongly calibrated often have many practical advantages such as transparency or ease of use [33].”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 6. Moderate calibration: a pragmatic guarantee for non-harmful decision-making
What the sources say
- Not checked yetcites [33] Marewski, 2012, Heuristic decision making in medicine, Dialogues Clin Neurosci source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means that at least part of the measurement error is systematic, and further thwarts the concept of the existence of a true model.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 5. Strong calibration: realistic or utopic?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Calibration is a pivotal aspect of model performance [2][3][4]”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 1. Introduction
What the sources say
- Not checked yetcites [2] Steyerberg, 2009
- Not checked yetcites [3] Kim, 2011, Probabilistic classifiers with high-dimensional data, Biostatistics source
- Not checked yetcites [4] Pepe, 2013, Methods for evaluating prediction performance of biomarkers and tests
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“measurement error of the predictors is in practice often ignored although it is known that this is a common phenomenon that can bias regression coefficients [28].”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 5. Strong calibration: realistic or utopic?
What the sources say
- Not checked yetcites [28] Fuller, 1987
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If the model's Net Benefit is lower than that of a default strategy, then using the model to support decision-making can be considered as clinically harmful.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > cex.d01
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“In previous simulations, the Hosmer-Lemeshow test showed such poor performance that it may not be recommended for routine use [7,37].”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 9. Statistical testing for calibration
What the sources say
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Net Benefit accounts for the different consequences of true and false positives through the risk threshold, i.e. the risk at which one is indifferent about classifying a patient as having the event (and providing treatment) or not.”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > cex.d01
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
Calibrated risk models are vital for valid decision support.
“Calibrated risk models are vital for valid decision support.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These approaches can be classified into three increasingly stringent levels, labelled as mean, weak, and moderate calibration. 52”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [52]
In the citing paper
“All discussed calibration measures are semi-proper, 52”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [52]
In the citing paper
“Estimated probabilities with too much spread are on average too close to 0 and 1 (slope < 1), whereas estimated probabilities with too little spread are on average too close to the prevalence (slope > 1).”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [52]
In the citing paper
“Previous recommendations were to include at least 100 to 200 individuals in the smallest outcome category.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > 4. Calibration, decision-making, and clinical utility
“Let us consider Example 9 (Figure 3C).The model presented is not calibrated in a weak sense.We derived decision curves to assess the clinical utility of (1) the original miscalibrated model, (2) a recalibrated model to ensure moderate calibration, and (3) the strongly calibrated true model (Figure 5).”
§ A calibration hierarchy for risk models was defined: from utopia to empirical data > cex.d01
“Figure: 5 Figure 5 . Figure 5. Decision curves for Example 8 to assess clinical usefulness.”
Citations
Cites
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Self-citationClinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Self-citationStefanski, L. A., Fuller, W. A., & Wiley, J. (1988) · Journal of Business and Economic Statistics · doi:10.2307/1391893 · Cited by 2,825
Dichotomizing continuous predictors in multiple regression: a bad idea
Royston, P., Altman, D. G., & Sauerbrei, W. (2006) · Statistics in Medicine · doi:10.1002/sim.2331 · Cited by 2,312
Rao, S. J. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s263 · Cited by 1,967
The number of subjects per variable required in linear regression analyses
Austin, P. C., & Steyerberg, E. W. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.12.014 · Cited by 1,145
Self-citationModel uncertainty, data mining and statistical inference
Chatfield, C. (1995) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2983440 · Cited by 1,103
Clinical Prediction Models: A Practical Approach to Development, Validation and Updating
Mann, C. J. H. (2009) · Kybernetes · doi:10.1108/k.2009.06738fae.002 · Cited by 794
Cited by
Sorted by citations
Items 201–300 of 918
Sabbagh, A., Washington, S. L., Tilki, D., Hong, J. C., Feng, J., Valdés, G., Chen, M., Wu, J., Huland, H., Graefen, M., et al. (2023) · European Urology Oncology · doi:10.1016/j.euo.2023.02.006 · Cited by 24
Muehlematter, U. J., Schweiger, L., Ferraro, D. A., Hermanns, T., Maurer, T., Heck, M. M., Rupp, N. J., Eiber, M., Rauscher, I., & Burger, I. A. (2023) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-023-06278-1 · Cited by 27
Aluvaala, J., Collins, P. G. S., Maina, M., Berkley, J. A., & English, M. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0649-6 · Cited by 23
Giardiello, D., Hauptmann, M., Steyerberg, E. W., Adank, M. A., Akdeniz, D., Blom, J., BLOMQVIST, C. P., Bojesen, S. E., Bolla, M. K., Brinkhuis, M., et al. (2020) · Breast Cancer Research and Treatment · doi:10.1007/s10549-020-05611-8 · Cited by 22
Self-citationA multidimensional ODE-based model of Alzheimer’s disease progression
Bossa, M. N., & Hichem Sahli, B. (2023) · Scientific Reports · doi:10.1038/s41598-023-29383-5 · Cited by 22
Gehringer, C. K., Martin, G. P., CALSTER, B. V., Hyrich, K. L., Verstappen, S. M. M., & Sergeant, J. C. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111481 · Cited by 22
Risk prediction models for frailty in older adults: A systematic review and critical appraisal
Kong, L., Li, Y., Lyu, Q., Liu, D., & Yang, J. (2025) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2025.105068 · Cited by 22
Martin, G. P., Riley, R. D., Ensor, J., & Grant, S. W. (2025) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezaf142 · Cited by 25
Engelhardt, E. G., Révész, D., Tamminga, H. J., Punt, C. J. A., Koopman, M., Onwuteaka-Philipsen, B. D., Steyerberg, E. W., Jansma, I. P., de Vet, H. C. W., & Coupé, V. M. H. (2017) · Clinical Colorectal Cancer · doi:10.1016/j.clcc.2017.06.007 · Cited by 22
Self-citationDong, W., Tse, E. T. Y., Mak, I. L., Wong, C. K. H., Wan, E. Y. F., Tang, E. H. M., Chin, W. Y., Bedford, L. E., Yu, E. Y. T., Ko, W. K. W., et al. (2022) · Journal of Diabetes Investigation · doi:10.1111/jdi.13790 · Cited by 26
Mining the contribution of intensive care clinical course to outcome after traumatic brain injury
Bhattacharyay, S., Caruso, P. F., Åkerlund, C. A. I., Wilson, L., Stevens, R. D., Menon, D. K., Steyerberg, E. W., Nelson, D., & Ercole, A. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00895-8 · Cited by 22
Self-citationBrAIST-Calc: Prediction of Individualized Benefit From Bracing for Adolescent Idiopathic Scoliosis
Dolan, L. A., Weinstein, S. L., Dobbs, M. B., Flynn, J. M. “., Green, D. W., Halsey, M. F., Hresko, M. T., Krengel, W. F., Mehlman, C. T., Milbrandt, T. A., et al. (2023) · Spine · doi:10.1097/brs.0000000000004879 · Cited by 23
Early Identification of Candidates for Epilepsy Surgery
Wissel, B. D., Greiner, H. M., Glauser, T. A., Pestian, J. P., Ficker, D. M., Cavitt, J. L., Estofan, L., Holland‐Bouley, K. D., Mangano, F. T., Szczesniak, R. D., et al. (2024) · Neurology · doi:10.1212/wnl.0000000000208048 · Cited by 22
Svinin, G. Е., Kutsenko, V. A., Shalnova, S. A., Yarovaya, E. B., Имаева, А. Э., Balanova, Y. А., Владимировна, К. А., Аркадьевна, М. Г., & Drapkina, O. M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0300974 · Cited by 22
Falconieri, N., CALSTER, B. V., Timmerman, D., & Wynants, L. (2020) · Biometrical Journal · doi:10.1002/bimj.201900075 · Cited by 22
Liang, S., Chau, J. P. C., Lo, S. H. S., Bai, L., Yao, L., & Choi, K. C. (2020) · Nursing in Critical Care · doi:10.1111/nicc.12550 · Cited by 22
Hagedoorn, N. N., Borensztajn, D., Nijman, R., Nieboer, D., Herberg, J. A., Balode, A., von Both, U., Carrol, E. D., Eleftheriou, I., Emonts, M., et al. (2020) · Archives of Disease in Childhood · doi:10.1136/archdischild-2020-319794 · Cited by 21
Self-citationEdlinger, M., van Smeden, M., Alber, H. F., Wanitschek, M. M., & CALSTER, B. V. (2021) · Statistics in Medicine · doi:10.1002/sim.9281 · Cited by 22
van der Ende, N. A. M., Kremers, F. C. C., van der Steen, W., Venema, E., Kappelhof, M., Majoie, C. B. L. M., Postma, A. A., Boiten, J., van den Wijngaard, I. R., van der Lugt, A., et al. (2023) · Stroke · doi:10.1161/strokeaha.122.040065 · Cited by 21
Granholm, A., Perner, A., Krag, M., Hjortrup, P. B., Haase, N., Holst, L. B., Marker, S., Collet, M. O., Jensen, A. K. G., & Møller, M. H. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-015339 · Cited by 20
Prognostic risk models for incident hypertension: A PRISMA systematic review and meta-analysis
Schjerven, F. E., Lindseth, F., & Steinsland, I. (2024) · PLoS ONE · doi:10.1371/journal.pone.0294148 · Cited by 24
van Amstel, R. B. E., Rademaker, E., Kennedy, J. N., Bos, L. D. J., Sengers, H. P., Butler, J. M., Bruse, N., Dongelmans, D. A., Kox, M., Vlaar, A. P. J., et al. (2025) · Critical Care · doi:10.1186/s13054-025-05256-3 · Cited by 20
Ren, C., Wu, C., Pan, Z., Wang, Q., & Li, Y. (2021) · Journal of Cardiothoracic Surgery · doi:10.1186/s13019-021-01450-w · Cited by 20
Statistical methods for validation of predictive models
Diniz, M. A. (2022) · Journal of Nuclear Cardiology · doi:10.1007/s12350-022-02994-7 · Cited by 22
Eyþórsson, E. S., Rögnvaldsson, S., Þorsteinsdóttir, S., Long, T. E., Reed, E. R., Sigurðardóttir, G. Á., Viðarsson, B., Onundarson, P. T., Agnarsson, B. A., Sigurðardóttir, M. S., et al. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2540 · Cited by 20
Machine Learning Predicts Femoral and Tibial Implant Size Mismatch for Total Knee Arthroplasty
Polce, E. M., Kunze, K. N., Paul, K. M., & Levine, B. R. (2021) · Arthroplasty Today · doi:10.1016/j.artd.2021.01.006 · Cited by 20
The Role of Artificial Intelligence in Chronic Liver Diseases and Liver Transplantation
Spann, A., Strauss, A. T., Davis, S. E., & Bhat, M. (2025) · Gastroenterology · doi:10.1053/j.gastro.2025.05.012 · Cited by 21
Yang, W., Zou, H., Wang, M., Zhang, Q., Li, S., & Liang, H. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e13200 · Cited by 20
Deforth, M. E., Heinze, G., & Held, U. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111539 · Cited by 19
External validation of the Simplified Mortality Score for the Intensive Care Unit (SMS‐ICU)
Granholm, A., Perner, A., Krag, M., Marker, S., Hjortrup, P. B., Haase, N., Holst, L. B., Collet, M. O., Jensen, A. K. G., & Møller, M. H. (2019) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.13422 · Cited by 19
Breast and Lung Effusion Survival Score Models
Molina, S., Martinez-Zayas, G., Sainz, P. V., Leung, C. H., Li, L., Grosu, H. B., Adachi, R., & Ost, D. E. (2021) · CHEST Journal · doi:10.1016/j.chest.2021.03.059 · Cited by 19
Nightingale Health Biobank Collaborative Group, Barrett, J. C., Esko, T., Fischer, K., Jostins-Dean, L., Jousilahti, P., Julkunen, H., Jääskeläinen, T., Kerimov, N., Kerminen, S., et al. (2023) · medRxiv · doi:10.1101/2023.06.09.23291213 · Cited by 19
Dynamic updating of clinical survival prediction models in a changing environment
Tanner, K., Keogh, R. H., Coupland, C. A. C., Hippisley‐Cox, J., & Diaz‐Ordaz, K. (2023) · Diagnostic and Prognostic Research · doi:10.1186/s41512-023-00163-z · Cited by 18
Krakowski, K., Oliver, D., Arribas, M., Stahl, D. R., & Fusar‐Poli, P. (2024) · Biological Psychiatry · doi:10.1016/j.biopsych.2024.05.022 · Cited by 18
Validation of Prediction Models for Pneumonia Among Children in the Emergency Department
Ramgopal, S., Lorenz, D. J., Navanandan, N., Cotter, J. M., Shah, S. S., Ruddy, R. M., Ambroggio, L., & Florin, T. A. (2022) · PEDIATRICS · doi:10.1542/peds.2021-055641 · Cited by 18
A scoring system to predict breast cancer mortality at 5 and 10 years
Paredes-Aracil, E., Palazón‐Bru, A., la Rosa, D. M. F.-D., Ots-Gutiérrez, J. R., Compañ, A., & Gil-Guillén, V. F. (2017) · Scientific Reports · doi:10.1038/s41598-017-00536-7 · Cited by 18
Estimating Throwing Speed in Handball Using a Wearable Device
Skejø, S. D., Bencke, J., Møller, M., & Sørensen, H. (2020) · Sensors · doi:10.3390/s20174925 · Cited by 18
Li, Q., Yao, X., & Échevin, D. (2020) · Value in Health · doi:10.1016/j.jval.2020.06.009 · Cited by 18
Comparison of dynamic updating strategies for clinical prediction models
Schnellinger, E. M., Yang, W. P., & Kimmel, S. E. (2021) · Diagnostic and Prognostic Research · doi:10.1186/s41512-021-00110-w · Cited by 18
Pavlou, M., Omar, R., & Ambler, G. (2024) · Biometrical Journal · doi:10.1002/bimj.202300245 · Cited by 17
Tiruneh, S. A., Rolnik, D. L., Teede, H. J., & Enticott, J. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110183 · Cited by 21
Accuracy of Risk Estimates from the iPrevent Breast Cancer Risk Assessment and Management Tool
Phillips, K., Liao, Y., Milne, R. L., MacInnis, R. J., Collins, I. M., Buchsbaum, R., Weideman, P. C., Bickerstaffe, A., Nesci, S., Chung, W. K., et al. (2019) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkz066 · Cited by 17
Uncertainty and the Value of Information in Risk Prediction Modeling
Sadatsafavi, M., Lee, T. Y., & Gustafson, P. (2022) · Medical Decision Making · doi:10.1177/0272989x221078789 · Cited by 17
Evaluation of different scoring systems for spinal metastases based on a Chinese cohort
Li, Z., Guo, L., Guo, B., Zhang, P., Wang, J., Wang, X., & Yao, W. (2022) · Cancer Medicine · doi:10.1002/cam4.5272 · Cited by 20
Zhong, G., Cheng, S., Zhou, M., Xie, J., Xu, Z., Lai, H., Yan, Y. C., Xie, Z., Zhou, J., Xie, X., et al. (2023) · The Spine Journal · doi:10.1016/j.spinee.2023.01.008 · Cited by 18
Hoogland, J., Efthimiou, O., Nguyen, T., & Debray, T. P. A. (2024) · Statistics in Medicine · doi:10.1002/sim.10186 · Cited by 17
Keene, D. J., Schlüssel, M. M., Thompson, J., Hagan, D. A., Williams, M. A., Byrne, C., Goodacre, S. W., Cooke, M. W., Gwilym, S. E., Hormbrey, P., et al. (2018) · Health Technology Assessment · doi:10.3310/hta22640 · Cited by 18
Ensor, J., Snell, K. I. E., Debray, T. P. A., Lambert, P. C., Look, M. P., Mamas, M. A., Moons, K. G. M., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8959 · Cited by 17
Reps, J., Williams, R. D., Schuemie, M. J., Ryan, P. B., & Rijnbeek, P. R. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01879-6 · Cited by 17
Mortality prediction models for community-dwelling older adults: A systematic review
Exmann, C. J., Kooijmans, E. C. M., Joling, K. J., Burchell, G. L., Hoogendijk, E. O., & van Hout, H. P. J. (2024) · Ageing Research Reviews · doi:10.1016/j.arr.2024.102525 · Cited by 17
Mortality Prediction in Hip Fracture Patients: Physician Assessment Versus Prognostic Models
Karres, J., Zwiers, R., Eerenberg, J. P., Vrouenraets, B. C., & Kerkhoffs, G. M. M. J. (2022) · Journal of Orthopaedic Trauma · doi:10.1097/bot.0000000000002412 · Cited by 16
Mari, T., Asgard, O., Henderson, J. R., Hewitt, D., Brown, C. A., Stančák, A., & Fallon, N. B. (2023) · Scientific Reports · doi:10.1038/s41598-022-27298-1 · Cited by 19
Kim, D. D., Wang, L., Lauren, B. N., Liu, J., Marklund, M., Lee, Y., Micha, R., Mozaffarian, D., & Wong, J. B. (2023) · Medical Decision Making · doi:10.1177/0272989x231196916 · Cited by 18
Hayden, J. A., Ogilvie, R., Stewart, S. A., French, S., Campbell, S., Magee, K. D., Slipp, P., Wells, G. A., & Stiell, I. G. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0047-8 · Cited by 16
External Validation of the Model of Thromboembolic Risk in Hypertrophic Cardiomyopathy Patients
He, S., Wang, Z., Cheem, T. H., Liao, H., Chen, X., & He, Y. (2019) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2019.05.035 · Cited by 16
Skalitzky, M. K., Gulbrandsen, T. R., Groot, O. Q., Karhade, A. V., Verlaan, J., Schwab, J. H., & Miller, B. J. (2021) · Journal of Surgical Oncology · doi:10.1002/jso.26708 · Cited by 15
Alejo, J. L., Mitchell, J., Chiang, T. P., Chang, A. Y., Abedon, A. T., Werbel, W. A., Boyarsky, B. J., Zeiser, L. B., Avery, R. K., Tobian, A. A. R., et al. (2022) · Transplantation · doi:10.1097/tp.0000000000004259 · Cited by 16
Fiori, C., Checcucci, E., Stura, I., Amparore, D., CILLIS, S. D., Piana, A., Granato, S., Volpi, G., Sica, M., Piramide, F., et al. (2022) · Prostate Cancer and Prostatic Diseases · doi:10.1038/s41391-022-00565-y · Cited by 15
Yu, F., Wu, P., Deng, H., Wu, J., Sun, S., Yu, H., Yang, J., Luo, X., He, J., Ma, X., et al. (2022) · Journal of Medical Internet Research · doi:10.2196/34126 · Cited by 16
Prediction of long-term mortality following hip fracture surgery: evaluation of three risk models
Karres, J., Eerenberg, J. P., Vrouenraets, B. C., & Kerkhoffs, G. M. M. J. (2022) · Archives of Orthopaedic and Trauma Surgery · doi:10.1007/s00402-022-04646-4 · Cited by 15
Neal, S. R., Fitzgerald, F., Chimhuya, S., Heys, M., Cortina‐Borja, M., & Chimhini, G. (2023) · Archives of Disease in Childhood · doi:10.1136/archdischild-2022-325158 · Cited by 15
Hu, Y., Lu, H. S., Ren, L., Yang, M., Shen, M., Huang, J., Huang, Q., & Fu, L. (2023) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2023.104546 · Cited by 15
Kyriacou, C., Ledger, A., Bobdiwala, S., Ayim, F., Kirk, E., Abughazza, O., Guha, S., Vathanan, V., Gould, D., Timmerman, D., et al. (2023) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.27515 · Cited by 15
de Vette, S. P. M., van Rijn–Dekker, M. I., den Bosch, L. V., Keijzer, K., Neh, H., Chu, H., Li, Y., Frederiks, M. L., van der Laan, H. P., Heukelom, J., et al. (2025) · Oral Oncology · doi:10.1016/j.oraloncology.2025.107224 · Cited by 15
Edlinger, M., Wanitschek, M. M., Dörler, J., Ulmer, H., Alber, H. F., & Steyerberg, E. W. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-014467 · Cited by 15
Self-citationImproving clinical management of COVID-19: the role of prediction models
Wynants, L., & Sotgiu, G. (2021) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(21)00006-0 · Cited by 15
Landolfo, C., Ceusters, J., Valentin, L., Froyman, W., Van Gorp, T., Heremans, R., Baert, T., Wouters, R., Vankerckhoven, A., Van Rompuy, A.-S., et al. (2024) · British Journal of Cancer · doi:10.1038/s41416-024-02578-x · Cited by 14
Gould, D. J., Bailey, J. A., Spelman, T. D., Bunzli, S., Dowsey, M. M., & Choong, P. (2023) · Arthroplasty · doi:10.1186/s42836-023-00186-3 · Cited by 14
Clinical Utility of Melanoma Sentinel Lymph Node Biopsy Nomograms
Drebin, H. M., Hosein, S., Kurtansky, N. R., Nadelmann, E. R., Moy, A. P., Ariyan, C. E., Bello, D. M., Brady, M. S., Coit, D. G., Marchetti, M. A., et al. (2023) · Journal of the American College of Surgeons · doi:10.1097/xcs.0000000000000886 · Cited by 14
Maddineni, S., Dizon, M. P., Muralidharan, V., Young, L. A., Sunwoo, J. B., Baik, F. M., & Swetter, S. M. (2024) · Annals of Surgical Oncology · doi:10.1245/s10434-023-14862-w · Cited by 15
Lapi, F., Elisa, B., Michieli, R., Alessandro, P., Cricelli, I., Giampiero, M., Bruno, F., Prieto‐Alhambra, D., Brandi, M. L., & Cricelli, C. A. (2017) · Calcified Tissue International · doi:10.1007/s00223-016-0230-7 · Cited by 14
Derivation and Validation of a Prediction Model for Venous Thromboembolism in Primary Care
Dentali, F., Fontanella, A., Cohen, A. T., Simonetti, M., Parretti, L., Marconi, E., Parretti, D., Gussoni, G., Campanini, M., Agnelli, G., et al. (2020) · Thrombosis and Haemostasis · doi:10.1055/s-0040-1701483 · Cited by 14
Risk Prediction Models in Perioperative Medicine: Methodological Considerations
Collins, P. G. S., Ma, J., Gerry, S., Ohuma, E. O., Odondi, L., Trivella, M., de Beyer, J. A., & Vazquez-Montes, M. (2016) · Current anesthesiology reports · doi:10.1007/s40140-016-0171-8 · Cited by 14
Dankers, F. J. W. M., Wijsman, R., Troost, E. G. C., Tissing-Tan, C. J. A., Kwint, M., Belderbos, J. S. A., Ruysscher, D. D., Hendriks, L. E. L., de Geus‐Oei, L., Rodwell, L., et al. (2018) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2018.07.021 · Cited by 14
Rospleszcz, S., Thorand, B., Gala, T. D. L. H., Meisinger, C., Holle, R., Köenig, W., Mansmann, U., & Peters, A. (2018) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2018-211102 · Cited by 14
Kocbek, S., Kocbek, P., Stožer, A., Zupanič, T., Groza, T., & Štiglic, G. (2018) · PeerJ · doi:10.7717/peerj.5765 · Cited by 14
Tran, A., Taljaard, M., Abdulaziz, K. E., Matar, M. M., Lampron, J., Steyerberg, E. W., & Vaillancourt, C. (2020) · Canadian Journal of Surgery · doi:10.1503/cjs.010619 · Cited by 14
Self-citationCho, J., Woo, S., Hwang, S. H., Kim, S., Lee, H., Hwang, J.-Y., Kim, J.-W., Kim, M. S., Smith, L., Lee, S., et al. (2025) · Diabetes Care · doi:10.2337/dc25-0355 · Cited by 16
Zhang, H., Chen, D., Shao, J., Zou, P., Cui, N., Tang, L., Wang, X., Wang, D., & Ye, Z. (2021) · Diabetes Metabolic Syndrome and Obesity · doi:10.2147/dmso.s316950 · Cited by 13
Aghakhani, A., Yousefi, M., & Yekaninejad, M. S. (2023) · Annals of Otology Rhinology & Laryngology · doi:10.1177/00034894231206902 · Cited by 15
Fu, Y., Feller, D., Koes, B. W., & Chiarotto, A. (2024) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2024.12081 · Cited by 16
Pascual, M. A., Vancraeynest, L., Timmerman, S., Ceusters, J., Ledger, A., Graupera, B., Rodríguez, I., Valero, B., Landolfo, C., Testa, A. C., et al. (2024) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.27642 · Cited by 16
Yao, M. W. M., Nguyen, E. T., Retzloff, M. G., Gago, L. A., Nichols, J. E., Payne, J. F., Ripps, B. A., Opsahl, M. S., Groll, J. M., Beesley, R. D., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-58744-z · Cited by 13
The simple 10-item PARC tool to predict childhood asthma - an external validation.
Pedersen, E. S. L., Spycher, B. D., de Jong, C. C. M., Halbeisen, F. S., Ramette, A. N., Gaillard, E. A., Granell, R., Henderson, A. J., & Kuehni, C. E. (2019) · doi:10.7892/boris.120546 · Cited by 13
Lokaj, P., Špinar, J., Špinarová, L., Málek, F., Ludka, O., Krejčí, J., Ošťádal, P., Vondráková, D., Lábr, K., Špinarová, M., et al. (2021) · PLoS ONE · doi:10.1371/journal.pone.0255271 · Cited by 13
Risk scores for predicting small for gestational age infants in Japan: The TMM birthree cohort study
Iwama, N., Obara, T., Ishikuro, M., Murakami, K., Ueno, F., Noda, A., Onuma, T., Matsuzaki, F., Hoshiai, T., Saito, M., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-12892-0 · Cited by 13
Machine Learning–Based Prediction of Masked Hypertension Among Children With Chronic Kidney Disease
Bae, S., Samuels, J., Flynn, J. T., Mitsnefes, M. M., Furth, S. L., Warady, B. A., Ng, D. K., & Study, F. T. C. (2022) · Hypertension · doi:10.1161/hypertensionaha.121.18794 · Cited by 13
McLernon, D. J., & Bhattacharya, S. (2022) · Best Practice & Research Clinical Obstetrics & Gynaecology · doi:10.1016/j.bpobgyn.2022.102309 · Cited by 13
Hasannejadasl, H., Roumen, C., van der Poel, H. G., Vanneste, B. G. L., van Roermund, J. G. H., Aben, K. K. H., Kalendralis, P., Osong, B., Kiemeney, L. A. L. M., van Oort, I. M., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0276815 · Cited by 13
Gao, S., Albu, E., Tuand, K., Cossey, V., Rademakers, F. E., CALSTER, B. V., & Wynants, L. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.07.019 · Cited by 14
Faisal, M., Howes, R., Steyerberg, E. W., Richardson, D., & Mohammed, M. A. (2016) · QJM · doi:10.1093/qjmed/hcw110 · Cited by 13
Self-citationA framework for evaluating predictive models
Tan, Y., Saffari, S. E., & Tan, N. C. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.08.005 · Cited by 12
Borges, A. L., Brito, M., Ambrósio, P., Condeço, R., Pinto, P., Ambrósio, B., Mahomed, F., Gama, J. M. R., Bernardo, M. J., Gouveia, A., et al. (2024) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.27641 · Cited by 12
Djulbegović, B., Boylan, A. M., Kolo, S. J., Scheurer, D. B., Anuskiewicz, S., Khaledi, F., Youkhana, K., Madgwick, S., Maharjan, N., & Hozo, I. (2024) · Blood Advances · doi:10.1182/bloodadvances.2024013166 · Cited by 12
Biomarker Panels for Discriminating Risk of CKD Progression in Children
Greenberg, J. H., Abraham, A. G., Xu, Y., Schelling, J. R., Coca, S. G., Schrauben, S. J., Wilson, F. P., Waikar, S. S., Vasan, R. S., Gutiérrez, O. M., et al. (2025) · Journal of the American Society of Nephrology · doi:10.1681/asn.0000000602 · Cited by 12
External validation of 12 existing survival prediction models for patients with spinal metastases
Bindels, B. J. J., Kuijten, R. H., Groot, O. Q., Huele, E. H., Gal, R., de Groot, M., van der Velden, J. M., Delawi, D., Schwab, J. H., Verkooijen, H. M., et al. (2025) · The Spine Journal · doi:10.1016/j.spinee.2025.01.014 · Cited by 12
Kuipers, H., Hoogwater, F. J. H., Holtman, G. A., Slangen, J. J. G., de Haas, R. J., & de Boer, M. T. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.109619 · Cited by 12
Early standardized clinical judgement for syncope diagnosis in the emergency department
de Lavallaz, J. D. F., Badertscher, P., Zimmermann, T., Nestelberger, T., Walter, J. E., Strebel, I., Coelho, C., Miró, Ò., Salgado, E., Christ, M., et al. (2021) · Journal of Internal Medicine · doi:10.1111/joim.13269 · Cited by 13
Zhang, H., Chen, D., Shao, J., Zou, P., Cui, N., Tang, L., Wang, D., & Ye, Z. (2021) · Diabetes Metabolic Syndrome and Obesity · doi:10.2147/dmso.s288881 · Cited by 12
Castro, V. M., Hart, K. L., Sacks, C. A., Murphy, S. N., Perlis, R. H., & McCoy, T. H. (2021) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2021.10.005 · Cited by 13
Metadata
- Authors
- Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W.
- Affiliations
- Erasmus MC (NL)
- Duke University (US)
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2016-06
- Volume
- 74
- Pages
- 167-176
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2225109326
- Cited by
- 910
- References
- 42
- Citations by year
- 2026: 192 · 2025: 99 · 2024: 118 · 2023: 93 · 2022: 125 · 2021: 91 · 2020: 58 · 2019: 49 · 2018: 35 · 2017: 30 · 2016: 19
- FWCI
- 44.92
- Type
- journal article
- Language
- en
- Open access
- green
- lirias.kuleuven.be
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Lirias (KU Leuven) (submittedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Funders
- Fonds Wetenschappelijk Onderzoek (G049312N)
- KU Leuven (C24/15/037)
- Vlaamse regering (G049312N, C24/15/037)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- risk model calibration, external validation, decision support, overfitting, simulation study
- MeSH terms
- Calibration, Computer Simulation, Decision Support Techniques, Humans, Risk, Reproducibility of Results, Models, Statistical, Bias, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
