Developing clinical prediction models: a step-by-step guide

May combine two claims.

A literature review showed limitations of existing prediction models for RRMS, including lack of internal validation, inadequate handling of missing data, and lack of assessment of clinical utility.

“Additionally, the review showed limitations of existing prediction models, including lack of internal validation, inadequate handling of missing data, and lack of assessment of clinical utility (step 1).”

May combine two claims.

The development data included 1752 observations from 935 patients followed up every two years, with 302 events observed.

“Data included a total of 1752 observations from 935 patients followed up every two years, with 302 events observed (step 5).”

• cites [120] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

A review of prediction models for cardiovascular disease identified 363 models and concluded that the usefulness of most models remains unclear owing to methodological shortcomings, incomplete presentation, and lack of external validation and model impact studies.

“a review of prediction models for cardiovascular disease identified 363 models19; the authors concluded that “the usefulness of most of the models remains unclear owing to methodological shortcomings, incomplete presentation, and lack of external validation and model impact studies.””

• cites [19] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Predictors not causally related to the outcome but strongly associated with it might still contribute to model performance, although they might generalise less well to different settings than causal factors.

“Predictors not causally related to the outcome but strongly associated with it might still contribute to model performance, although they might generalise less well to different settings than causal factors.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The imputation model might involve auxiliary variables associated with missing data, which can enhance the effectiveness of the imputations.

“Additionally, the imputation model might involve auxiliary variables associated with missing data, which can enhance the effectiveness of the imputations.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Patients receiving a new treatment might be more likely to miss follow-up visits if the treatment was successful, leading to missing data.

“patients receiving a new treatment might be more likely to miss follow-up visits if the treatment was successful, leading to missing data.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Common pitfalls in prediction modelling include inappropriate categorising of continuous outcomes or predictors, data driven cut-off points, univariable selection methods, overfitting, and lack of attention to missing data and a sound assessment of performance and clinical benefit.

“Common pitfalls include inappropriate categorising of continuous outcomes or predictors, data driven cut-off points, univariable selection methods, overfitting, and lack of attention to missing data and a sound assessment of performance and clinical benefit”

May combine two claims.

If no auxiliary variables were used in the imputations, people with imputed outcomes can be removed, and the model can be developed based only on people with observed outcomes.

“If no auxiliary variables were used in the imputations, people with imputed outcomes can be removed, and the model can be developed based only on people with observed outcomes.57”

• cites [57] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In practice, developing prediction models using existing data from cohort studies or other data not collected explicitly for this purpose is much more common.

“In practice, developing prediction models using existing data from cohort studies or other data not collected explicitly for this purpose is much more common.”

Classified as probably not a substantial claim. May combine two claims.

Multiple imputation is the approach usually recommended to handle missing data during model development, and appropriately accounts for missing data uncertainty.

“Multiple imputation is the approach usually recommended to handle missing data during model development, and appropriately accounts for missing data uncertainty.56”

• cites [56] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Dichotomising or categorising continuous predictors reduces information and diminishes statistical power.

“Dichotomising or categorising continuous predictors reduces information and diminishes statistical power and should be avoided.41 42”

• cites [41] sourceNot checked yet

  Not checked in the free run.

• cites [42] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Linear regression for continuous outcomes, logistic regression for binary outcomes, and Cox or simple parametric models for survival outcomes are the usual starting points in modelling.

“Linear regression for continuous outcomes, logistic regression for binary outcomes, and Cox or simple parametric models for survival outcomes are the usual starting points in modelling.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Additional parameters will be needed for including nonlinear terms or interactions among the predictors in the list.

“Note that additional parameters will be needed for including nonlinear terms or interactions among the predictors in the list.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A very simple model or a model based on covariates that are not associated with the outcome will perform poorly in the data used to develop it and in new data (underfitting).

“A very simple model or a model based on covariates that are not associated with the outcome will perform poorly in the data used to develop it and in new data; this scenario is called underfitting.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

If imputation incorporates auxiliary variables, including those with imputed outcomes in the model development is advisable.

“However, if imputation incorporates auxiliary variables, including those with imputed outcomes in the model development is advisable.58”

• cites [58] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Single imputation can perform well, although multiple imputation tends to be more consistent and stable.

“Sisk and colleagues showed that single imputation can perform well, although multiple imputation tends to be more consistent and stable.59”

• cites [59] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The quality of trial data will generally be high, but models could have limited generalisability because trial participants might not represent the patients seen in clinical practice.

“The quality of trial data will generally be high, but models could have limited generalisability because trial participants might not represent the patients seen in clinical practice.”

May combine two claims.

Registry data have the main advantage of relatively large sample size and representativeness.

“Registry data offer a simple, low cost alternative; their main advantage is the relatively large sample size and representativeness.”

• cites [47] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

The concepts described and the steps proposed largely apply to statistical and machine learning models.

“The concepts we describe and the steps we propose largely apply to statistical and machine learning models.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

If the sample size is large enough, models can include nonlinear terms for continuous predictors or interactions between predictors.

“If the sample size is large enough (see step 6), models can include nonlinear terms for continuous predictors or interactions between predictors.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In some cases, thresholds are chosen after exploring various cut-off points and opting for those that fit the data best or yield statistically significant results.

“In some cases, thresholds are chosen after exploring various cut-off points and opting for those that fit the data best or yield statistically significant results.”

Probably not a substantial claim: may not pass S4 (based).

Prioritising predictors with proven or suspected causal relationships might increase the model's generalisability.

“Predictors with proven or suspected causal relationships with the outcome should be prioritised for inclusion; this approach might increase the model's generalisability.”

May combine two claims.

Machine learning models might add value if there are strong nonlinearities and interactions between predictors, although they are not immune to biases.

“These strategies might add value if there are strong nonlinearities and interactions between predictors, although they are not immune to biases.64”

• cites [64] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Sample size calculations for machine learning models are considerably more complex and might require simulations.

“Sample size calculations for such models are considerably more complex and might require simulations.54”

• cites [54] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Imputation methods are not a panacea and might fail, typically when the tendency of the outcome to be missing correlates with the outcome itself.

“Ιmputation methods are not a panacea and might fail, typically when the tendency of the outcome to be missing correlates with the outcome itself.”

May combine two claims.

In practice, overfitting is more common than underfitting because datasets are often small and have few events, and there is the temptation to create models with the best (apparent) performance.

“In practice, overfitting is more common than underfitting because datasets are often small and have few events, and there is the temptation to create models with the best (apparent) performance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A smooth calibration line can assess calibration locally, indicating areas where the model underestimates or overestimates the outcome.

“A smooth calibration line can assess calibration locally—ie, it can indicate areas where the model underestimates or overestimates the outcome.”

May combine two claims.

Using the same dataset for development and performance assessment (apparent validation) might overestimate model performance, leading to optimistic assessments.

“However, this strategy might overestimate the model’s performance (fig 1); that is, it might lead to erroneous (optimistic) assessments.”

May combine two claims.

Penalisation is closely related to the bias-variance trade-off and aims to bring the model closer to the sweet spot of the bias-variance trade-off curve, where model performance in new data is maximised.

“Penalisation is closely related to the bias-variance trade-off depicted in figure 1, and is a method aiming to bring the model closer to the sweet spot of the bias-variance trade-off curve, where model performance in new data is maximised”

• cites [73] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Rubin's rule might not apply to machine learning methods because the m models could have different architectures.

“Rubin’s rule might not apply to machine learning methods because the m models could have different architectures.”

May combine two claims.

Univariable selection methods do not consider the association between predictors and could lead to loss of valuable information.

“These methods do not consider the association between predictors and could lead to loss of valuable information.”

• cites 55 66Not checked yet

  Not checked in the free run.

May combine two claims.

The split sample approach is problematic because it wastes data and decreases statistical power.

“However, this method is problematic because it wastes data and decreases statistical power.”

• cites 55 87Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

For a perfectly calibrated model, the observed versus predicted line is on the diagonal; for an overfit model, the calibration line is above the diagonal; for an underfit model, below.

“For a perfectly calibrated model, this line is on the diagonal; for an overfit (underfit) model, the calibration line is above (below) the diagonal.”

Probably not a substantial claim: may not pass S3 (load-bearing).

If there are m imputed datasets, m different models will be developed for each modelling strategy.

“Consequently, if there are m imputed datasets, m different models will be developed for each modelling strategy.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A method for combining the m models is to use them to make predictions for the new person and then average these m predictions, conceptually similar to stacking in machine learning.

“Another method for combining the m models is to use them to make predictions for the new person and then average these m predictions,79 a procedure conceptually similar to stacking in machine learning.”

• cites [79] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

A well calibrated model might have poor discrimination.

“Moreover, a well calibrated model might have poor discrimination.”

May combine two claims.

Stepwise methods for variable selection might lead to bias in estimation and worse predictive performance.

“Again, they are not recommended because they might lead to bias in estimation and worse predictive performance.”

• cites 55 67 68Not checked yet

  Not checked in the free run.

May combine two claims.

The time point used for splitting data in temporal validation is generally arbitrary, and older data might not reflect current patient characteristics or health care.

“However, the time point used for splitting the data will generally be arbitrary and older data might not reflect current patient characteristics or health care.”

May combine two claims.

Smooth calibration curves can be used to assess performance across the full range of predicted risks, and additional calibration metrics have been proposed.

“Smooth calibration curves can also be used to assess performance of the model across the full range of predicted risks, while additional calibration metrics have also been proposed.”

• cites 85 86Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

For continuous outcomes, observed outcomes can be directly compared with predictions because they are on the same scale.

“For continuous outcomes like blood pressure this is straightforward: observed outcomes can be directly compared with predictions because they are on the same scale.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Looking at estimated coefficients in (generalised) linear regression models is a simple way to assess the importance of different predictors.

“Looking at estimated coefficients in (generalised) linear regression models is a simple way to assess the importance of different predictors.”

May combine two claims.

One successful external validation does not establish transportability to many other settings, and requiring external validation might lead to selective reporting of validation data.

“One successful external validation, however, does not establish transportability to many other settings, while such a requirement might lead to the selective reporting of validation data.100”

• cites [100] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

EuroSCORE is a model developed in 1999 for predicting mortality in hospital for patients undergoing cardiac surgery.

“For example, EuroSCORE is a model developed in 1999 for predicting mortality in hospital for patients undergoing cardiac surgery.99”

• cites [99] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The inclusion of external validation in model development is a topic of debate, with certain journals mandating it for publication.

“The inclusion of external validation in model development is a topic of debate, with certain journals mandating it for publication.88 100”

• cites 88 100Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Net benefit brings the benefits and harms of a decision strategy on the same scale so they can be compared.

“Net benefit brings the benefits and harms of a decision strategy (eg, to decide for or against treatment based on a prediction model) on the same scale so they can be compared.104”

• cites [104] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

In prediction modelling, the primary focus is typically not on evaluating the importance of individual predictors; rather, the goal is to optimise the model's overall predictive performance.

“In prediction modelling, the primary focus is typically not on evaluating the importance of individual predictors; rather, the goal is to optimise the model’s overall predictive performance.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some predictors might be modifiable, raising the possibility that they could play a part in prevention if their association with the outcome is causal.

“Also, some predictors might be modifiable, raising the possibility that they could play a part in prevention if their association with the outcome is causal.”

Probably not a substantial claim: may not pass S3 (load-bearing).

When the assumptions of linear regression are not met, for example, when there is collinearity, estimates of coefficients might be unreliable.

“However, when the assumptions of linear regression are not met, for example, when there is collinearity, these estimates might be unreliable.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Identifying influential predictors might be of interest, for example, when evaluating the potential inclusion of a new biomarker as a routine measurement.

“Nevertheless, identifying influential predictors might be of interest, for example, when evaluating the potential inclusion of a new biomarker as a routine measurement.”

Probably not a substantial claim: may not pass S3 (load-bearing).

NICE in the UK recommends cholesterol lowering treatment if the predicted 10 year risk of myocardial infarction or stroke is 10% or higher based on the QRISK3 risk calculator.

“For example, the National Institute for Health and Care Excellence (NICE) in the UK recommends cholesterol lowering treatment if the predicted 10 year risk of myocardial infarction or stroke is 10% or higher (the cut-off threshold probability) based on the QRISK3 risk calculator.60 105”

• cites 60 105Not checked yet

  Not checked in the free run.

• AssumptionThe authors state this as an assumption: The benefit of treating one patient who would experience a cardiovascular event over 10 years outweighs the harms and costs incurred by treating another nine people who will not benefit.
• AssumptionThe authors state this as an assumption: The harm associated with not treating the one patient who would develop the event is nine times greater than the consequences of treating a patient who does not need it.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Bootstrapping first and then imputing might be computationally difficult.

“However, this strategy might be computationally difficult.”

May combine two claims.

Using data from 2001 to 2011, EuroSCORE was shown to consistently overestimate mortality and its calibration deteriorated over time.

“Using data from 2001 to 2011, EuroSCORE was shown to consistently overestimate mortality and its calibration deteriorated over time.100”

• cites [100] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In practice, bootstrapping often needs to be combined with multiple imputation.

“In practice, we often need to combine bootstrapping with multiple imputation.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

While an accurate prediction model can be valuable in counselling patients on likely outcomes, determining its utility in guiding decisions is less straightforward.

“While an accurate prediction model can be valuable in counselling patients on likely outcomes, determining its utility in guiding decisions is less straightforward.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The choice of decision threshold can be subjective, and the range of sensible thresholds depends on settings, conditions, available diagnostic tests or treatments, and patient preferences.

“The choice of decision threshold can be subjective, and the range of sensible thresholds will depend on the settings, conditions, available diagnostic tests or treatments, and patient preferences.”

May combine two claims.

Bootstrapping generally leads to more stable and less biased results.

“Bootstrapping generally leads to more stable and less biased results,93 and is therefore recommended for internal validation.”

• cites [93] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Prediction models for patients with HIV were developed based on data from treatment programmes in Côte d'Ivoire, South Africa, and Malawi and validated using leave-one-country-out cross validation.

“For example, prediction models for patients with HIV were developed based on data from treatment programmes in Côte d’Ivoire, South Africa, and Malawi and validated using leave-one-country-out cross validation.97”

• cites [97] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

A prediction model might strongly discriminate and be well calibrated, but its value depends on how we intend to use it in clinical practice.

“A prediction model might strongly discriminate and be well calibrated, but its value depends on how we intend to use it in clinical practice.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Riley and Collins described how to assess the stability of clinical prediction models during the model development phase using a bootstrap approach.

“Riley and Collins described how to assess the stability of clinical prediction models during the model development phase using a bootstrap approach.94”

• cites [94] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

A common scenario is when a model's discrimination is adequate in new settings and fairly stable over time, but calibration is suboptimal across settings or deteriorates over time (calibration drift).

“A common scenario is when a model’s discrimination is adequate in new settings and fairly stable over time, but calibration is suboptimal across settings or deteriorates over time (calibration drift).98”

• cites [98] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The lower the decision threshold, the more unnecessary tests or interventions we are willing to accept.

“The lower the threshold, the more unnecessary tests or interventions we are willing to accept.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Vickers and colleagues provide a step-by-step guide to interpreting decision curve analysis, and a website with a software tutorial and other resources.

“Vickers and colleagues provide a helpful step-by-step guide to interpreting decision curve analysis, and a website with a software tutorial and other resources.107”

• cites [107] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

If publicly accessible, online calculators might be misused by people for whom they are not intended, or if the model fails to show any clinical value.

“However, if publicly accessible, calculators might be misused by people for whom they are not intended, or if the model fails to show any clinical value (eg, in a subsequent external validation).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Regression coefficients were penalised through a Laplace prior distribution to address possible overfitting.

“Regression coefficients were penalised through a Laplace prior distribution to address possible overfitting (step 8).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Bootstrapping might not be computationally feasible for some machine learning approaches.

“Further, bootstrapping, which we recommended as the method of choice for internal validation, might not be computationally feasible for some machine learning approaches.”

May combine two claims.

Fitting the model with and without a predictor and noting the reduction in model performance can assess predictor importance; omitting more important predictors leads to a larger reduction in performance.

“Another method to assess the importance of a predictor, also applicable to machine learning models, is to fit the model with and without this predictor and note the reduction in model performance; omitting more important predictors will lead to a larger reduction in performance.”

May combine two claims.

Sample size calculations indicated a minimum sample of 2082 patients, which is larger than the available sample, raising concerns about possible overfitting.

“Sample size calculations50 indicated a minimum sample of 2082 patients, which is larger than the available sample, raising concerns about possible overfitting issues (step 6).”

• cites [50] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Simpler approaches for making models available are easier to use, for example, on ward rounds, but might require model simplification by removing some predictors or categorising continuous variables.

“Simpler approaches are easier to use, for example, on ward rounds, but might require model simplification by removing some predictors or categorising continuous variables.”

May combine two claims.

The model's discriminatory ability is comparable to or exceeding previous RRMS models.

“comparable to or exceeding previous RRMS models (steps 9 and 10)”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Bayesian logistic mixed effects prediction model used in the example accounted for multiple observations within patients.

“A Bayesian logistic mixed effects prediction model was developed, which accounted for several observations within patients.”

Classified as probably not a substantial claim.

Both performance assessments were corrected for optimism through a bootstrap validation procedure with 500 bootstrap samples created for each imputed dataset.

“Both assessments were corrected for optimism through a bootstrap validation procedure (described in box 2), with 500 bootstrap samples created for each imputed dataset.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Multicollinearity does not threaten a model's predictive performance, only the interpretation of the coefficients.

“However, note that multicollinearity does not threaten a model's predictive performance, just at the interpretation of the coefficients.”

Probably not a substantial claim: may not pass S3 (load-bearing).

More advanced methods to assess predictor importance include the permutation importance algorithm and SHAP (Shapley additive explanations).

“More advanced methods include the permutation importance algorithm109 and SHAP (Shapley additive explanations)110; we do not discuss these here.”

• cites 109 110Not checked yet

  Not checked in the free run.