Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The ADNEX model had an AUROC of 0.91 (95% CI 0.89-0.93) and an AUPRC of 0.89 (95% CI 0.86-0.91).

“The ADNEX model had an AUROC of 0 ⋅ 91 (95% CI 0 ⋅ 89-0 ⋅ 93) and an AUPRC of 0 ⋅ 89 (95% CI 0 ⋅ 86-0 ⋅ 91).”

May combine two claims.

The calibration intercept of the ADNEX model was 0.81 (95% CI 0.62-1.01), suggesting underestimation of probabilities on average (intercept > 0).

“The calibration intercept of the ADNEX model was 0 ⋅ 81 (95% CI 0 ⋅ 62-1 ⋅ 01), suggesting underestimation of probabilities on average (intercept > 0).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Ignoring sensitivity values below 0.8, pAUROC was 0.14 (95% CI 0.13-0.15).

“Ignoring sensitivity values below 0 ⋅ 8, pAUROC was 0 ⋅ 14 (95% CI 0 ⋅ 13-0 ⋅ 15).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Five of six participating centres in the validation study were tertiary care centres, resulting in a high prevalence of malignancy (49%).

“A probable reason is that fi ve of six participating centres in the validation study were tertiary care centres, resulting in a high prevalence of malignancy (49%).”

May combine two claims.

17 measures showed both characteristics, 14 showed one, and one (F1 score) showed neither.

“17 measures showed both characteristics, 14 showed one, and one (F1 score) showed neither.”

May combine two claims.

Some partial classification measures improved (sensitivity and NPV), whereas others worsened (specificity and PPV) after recalibration.

“Some partial classification measures improved (sensitivity and NPV), whereas others worsened (specificity and PPV).”

May combine two claims.

At a threshold of 10%, the ADNEX model showed a classification accuracy of 0.79 (95% CI 0.77-0.82), F1 score of 0.82 (0.79-0.84), and MCC of 0.63 (0.58-0.67).

“At a threshold of 10%, the ADNEX model showed a classification accuracy of 0 ⋅ 79 (95% CI 0 ⋅ 77-0 ⋅ 82), F1 score of 0 ⋅ 82 (0 ⋅ 79-0 ⋅ 84), and MCC of 0 ⋅ 63 (0 ⋅ 58-0 ⋅ 67) (table 1).”

May combine two claims.

At the commonly recommended threshold of 10%, the ADNEX model classified 578 patients as high risk, of whom 414 had a malignant tumour (true positive) and 164 had a benign tumour (false positive); the remaining 316 patients were classified as low risk, of whom 296 had a benign tumour (true negative) and 20 had a malignant tumour (false negative).

“At the commonly recommended threshold of 10% in our case study, 23 ADNEX classified 578 patients as high risk, of whom 414 had a malignant tumour (true positive) and 164 had a benign tumour (false positive). The remaining 316 patients were classified as low risk, of whom 296 had a benign tumour (true negative) and 20 had a malignant tumour (false negative).”

• cites [23] Timmerman, 2021, ESGO/ISUOG/IOTA/ESGE consensus statement on preoperative diagnosis of ovarian tumors, Ultrasound Obstet Gynecol sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Calibration is an active area of research focusing on aspects such as strong calibration, quantifying the degree of miscalibration, and uncertainty.

“For example, calibration is an active area of research focusing on aspects such as strong calibration, quantifying the degree of miscalibration, and uncertainty.”

• cites [62] Arrieta-Ibarra, 2022, Metrics of calibration for probabilistic predictions, J Mach Learn ResNot checked yet

  Not checked in the free run.

• cites [113] Perez-Lebel, 2022, Beyond calibration: estimating the grouping loss of modern neural networks, arXivNot checked yet

  Not checked in the free run.

• cites [114] Kompa, 2021, Second opinion needed: communicating uncertainty in medical machine learning, NPJ Digit Med sourceNot checked yet

  Not checked in the free run.

• cites [115] Riley, 2023, Stability of clinical prediction models developed using statistical or machine learning methods, Biom J sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

If misclassification costs influence a performance measure in an implicit or ad hoc way, the measure neither assesses statistical performance nor adequately evaluates the quality of decisions for clinical practice.

“If misclassification costs influence a performance measure in an implicit or ad hoc way, the measure neither assesses statistical performance nor adequately evaluates the quality of decisions for clinical practice.”

Classified as probably not a substantial claim. May combine two claims.

Discrimination reflects relative performance; it does not matter how high or low the estimated probabilities are, only whether they allow discrimination between individuals with versus without the event.

“Discrimination reflects relative performance; ie, it does not matter how high or low the estimated probabilities are, only whether they allow to discriminate between individuals with versus without the event.”

Classified as probably not a substantial claim.

Classification performance is influenced by discrimination and calibration performance.

“Classification performance is influenced by discrimination and calibration performance.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Using statistical arguments to set a decision threshold is inconsistent with decision theory and detached from practical use by clinicians.

“Using statistical arguments to set a decision threshold is inconsistent with decision theory and detached from practical use by clinicians.”

• AssumptionRests on an assumption: The primary aim of most predictive AI models in medical practice is to support subsequent decision making.

May combine two claims.

Statistical performance measures are essential for model evaluation but cannot be used to find out whether a model should be used in practice.

“Statistical performance measures are essential for model evaluation but cannot be used to fi nd out whether a model should be used in practice: it is not appropriate to cite, for example, good discrimination and calibration or conclude that a model can be used to aid decisions about ovarian surgery.”

Classified as probably not a substantial claim.

Clinical utility goes one step further by explicitly incorporating misclassification costs when evaluating classifications of individuals into low-risk and high-risk groups.

“The fi fth domain, clinical utility, goes one step further by explicitly incorporating misclassification costs when evaluating classifications of individuals into low-risk and high-risk groups.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The first three performance domains evaluate performance based on probability estimates.

“Among these domains, the fi rst three evaluate performance based on probability estimates (Appendix p 2).”

Classified as probably not a substantial claim. May combine two claims.

When an incorrect model can have a better expected value than the correct model, the measure is termed improper and cannot be trusted.

“When an incorrect model can have a better expected value than the correct model, the measure is termed improper and cannot be trusted.”

Classified as probably not a substantial claim. May combine two claims.

The importance of properness is that a proper measure cannot be fooled: in expectation, the correct model cannot be outperformed by an incorrect one.

“The importance of properness is that a proper measure cannot be fooled: in expectation, the correct model cannot be outperformed by an incorrect one.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The fourth and fifth performance domains require a threshold on the estimated risk of the event to classify individuals into two mutually exclusive groups.

“The fourth and fi fth performance domains require a threshold on the estimated risk of the event to classify individuals into two mutually exclusive groups: low-risk (estimated risk below the threshold) and high-risk (estimated risk equal to or above the threshold) groups.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Measures that do not possess the first characteristic cannot be trusted, whereas measures not possessing the second are equivocal.

“Measures that do not possess the fi rst characteristic cannot be trusted, whereas measures not possessing the second are equivocal.”

May combine two claims.

Poorly performing predictive AI models intended for medical practice are misleading and may lead to wrong clinical decisions that can be detrimental to patients and increase financial costs.

“poorly performing models might lead to wrong clinical decisions that can be detrimental to patients and increase fi nancial costs. 5”

• cites [5] Kamran, 2024, Evaluation of sepsis prediction models before onset of treatment, NEJM AI sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Calibration reflects absolute performance by evaluating whether estimates are too high or too low.

“Calibration reflects absolute performance by evaluating whether estimates are too high or too low.”

May combine two claims.

Models can have good discrimination but poor calibration, and vice versa.

“Models can therefore have good discrimination but poor calibration, and vice versa.”

Classified as probably not a substantial claim.

The definition of discrimination implies that discrimination measures should rely only on the ranks of the estimated probabilities in the dataset.

“The definition of discrimination implies that discrimination measures should rely only on the ranks of the estimated probabilities in the dataset. 36”

• cites [36] Berrar, 2012, Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them), Brief Bioinform sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Research on the quantification of a fourth level, strong calibration, is ongoing.

“Research on the quantification of a fourth level, strong calibration, is ongoing.”

Classified as probably not a substantial claim.

A decision threshold of 0.1 (10%) for the probability of malignancy is often recommended.

“A decision threshold of 0 ⋅ 1 (10%) for the probability of malignancy is often recommended. 23”

• cites [23] Timmerman, 2021, ESGO/ISUOG/IOTA/ESGE consensus statement on preoperative diagnosis of ovarian tumors, Ultrasound Obstet Gynecol sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: The primary aim of most predictive AI models in medical practice is to support subsequent decision making.

Classified as probably not a substantial claim. May combine two claims.

Discrimination measures are not meant to reflect differential misclassification costs, and class imbalance should not be conflated with misclassification costs or medical relevance.

“Discrimination measures are not meant to reflect differential misclassification costs, and class imbalance should not be conflated with misclassification costs or medical relevance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

ADNEX was developed on data from 5909 individuals recruited between 1999 and 2012 in 24 secondary and tertiary care centres from 10 countries.

“ADNEX was developed on the data from 5909 individuals recruited between 1999 and 2012 in 24 secondary and tertiary care centres from 10 countries (Italy, Belgium, Sweden, Czech Republic, Poland, France, United Kingdom, China, Spain, and Canada).”

• cites [31] Van Calster, 2014, Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study, BMJ sourceNot checked yet

  Not checked in the free run.

May combine two claims.

ADNEX can inform decisions regarding type of surgery (advanced vs conservative) for patients examined at an oncology centre or regarding referral to an oncology centre for patients examined elsewhere.

“The model can inform decisions regarding the type of surgery (advanced vs conservative) for patients examined at an oncology centre or regarding referral to an oncology centre for patients examined elsewhere. 32”

• cites [32] Landolfo, 2024, Comparison of the ADNEX and ROMA risk prediction models for the diagnosis of ovarian cancer: a multicentre external validation in patients who underwent surgery, Br J Cancer sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Estimated probabilities with too much spread are on average too close to 0 and 1 (calibration slope < 1), whereas estimated probabilities with too little spread are on average too close to the prevalence (calibration slope > 1).

“Estimated probabilities with too much spread are on average too close to 0 and 1 (slope &lt; 1), whereas estimated probabilities with too little spread are on average too close to the prevalence (slope &gt; 1).”

• cites [52] Van Calster, 2016, A calibration hierarchy for risk models was defined: from utopia to empirical data, J Clin Epidemiol sourceNot checked yet

  Not checked in the free run.

• cites [53] Stevens, 2020, Validation of clinical prediction models: what does the “calibration slope” really measure?, J Clin Epidemiol sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Logistic recalibration is a rank-preserving method, and the ranking of patients based on the estimated probability of malignancy remains the same before and after updating.

“This method is thus a rank-preserving method, and the ranking of patients based on the estimated probability of malignancy remains the same before and after updating.”

Probably not a substantial claim: may not pass S2 (checkable).

The weight of the consequences of using a model to support a decision might vary by the nature and effects of the intervention, the health-care system, or by clinician and patient.

“The weight of these consequences might vary by the nature and effects of the intervention, the health-care system, or by clinician and patient.”

• AssumptionRests on an assumption: The primary aim of most predictive AI models in medical practice is to support subsequent decision making.

May combine two claims.

ECE, ECI, and ICI depend on the grouping or smoothing methods used and have issues with statistical consistency.

“Further, ECE, ECI, and ICI depend on the grouping or smoothing methods used and have issues with statistical consistency.”

• cites [62] Arrieta-Ibarra, 2022, Metrics of calibration for probabilistic predictions, J Mach Learn ResNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

It is not appropriate to cite good discrimination and calibration or conclude that a model can be used to aid decisions about ovarian surgery.

“it is not appropriate to cite, for example, good discrimination and calibration or conclude that a model can be used to aid decisions about ovarian surgery.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There were 434 women with a malignant tumour (prevalence 49%).

“There were 434 women with a malignant tumour (prevalence 49%). 32”

• cites [32] Landolfo, 2024, Comparison of the ADNEX and ROMA risk prediction models for the diagnosis of ovarian cancer: a multicentre external validation in patients who underwent surgery, Br J Cancer sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Often, a threshold is chosen by optimising a statistical measure such as the Youden index.

“Often, however, a threshold is chosen by optimising a statistical measure such as the Youden index (sensitivity + specificity -1).”

• AssumptionRests on an assumption: The primary aim of most predictive AI models in medical practice is to support subsequent decision making.

May combine two claims.

Discrimination slope and mean absolute prediction error are improper.

“discrimination slope and mean absolute prediction error are improper.”

• cites [26] BujaNot checked yet

  Not checked in the free run.

• cites [27] Gneiting, 2007, Strictly proper scoring rules, prediction, and estimation, J Am Stat Assoc sourceNot checked yet

  Not checked in the free run.

• cites [71] Van Houwelingen, 2012Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Although true negatives might be highly irrelevant for selected non-medical applications, such errors are generally important in medical applications.

“Although true negatives might be highly irrelevant for selected non-medical applications, such errors are generally important in medical applications.”

May combine two claims.

Although discriminatory ability is essential for predictive AI, AUROC alone cannot be used to identify a model as good or useful.

“Nevertheless, although discriminatory ability is essential for predictive AI, AUROC alone cannot be used to identify a model as good or useful. 51”

• cites [51] de Hond, 2022, Interpreting area under the receiver operating characteristic curve, Lancet Digit Health sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Logistic recalibration is similar to Platt scaling, a well known method in machine learning to improve the calibration of predictions.

“This method is similar to Platt scaling, a well known method in machine learning to improve the calibration of predictions. 34,35”

• cites [34] Ojeda, 2023, Calibrating machine learning approaches for probability estimation: a comprehensive comparison, Stat Med sourceNot checked yet

  Not checked in the free run.

• cites [35] Platt, 1999, Probabilistic outputs for support vector machines and comparison to regularized likelihood methodsNot checked yet

  Not checked in the free run.

May combine two claims.

The TransIOTA study externally validated ADNEX for its ability to distinguish benign from malignant tumours, using data from 894 women recruited between 2015 and 2019 in one secondary and five tertiary care centres in four countries.

“The TransIOTA study externally validated ADNEX for its ability to distinguish benign from malignant tumours, using data from 894 women recruited between 2015 and 2019 in one secondary and five tertiary care centres in four countries (Belgium, Italy, Czech Republic, and United Kingdom).”

• cites [32] Landolfo, 2024, Comparison of the ADNEX and ROMA risk prediction models for the diagnosis of ovarian cancer: a multicentre external validation in patients who underwent surgery, Br J Cancer sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

In biomedical applications, the consequences of a false negative are almost always different from the consequences of a false positive.

“In biomedical applications, the consequences of a false negative (for instance, not referring a woman with ovarian malignancy for advanced surgery) are almost always different from the consequences of a false positive (referring a woman with a benign tumour for advanced surgery).”

May combine two claims.

AUPRC and pAUROC mix statistical performance with aspects of clinical utility without following decision-analytical principles.

“AUPRC and pAUROC mix statistical performance with aspects of clinical utility without following decision-analytical principles.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For binary outcomes, the C-statistic is equal to area under the receiver operating characteristic curve (AUROC).

“For binary outcomes, the C-statistic is equal to area under the receiver operating characteristic curve (AUROC). 37,38”

• cites [37] Bradley, 1997, The use of the area under the ROC curve in the evaluation of machine learning algorithms, Pattern Recognit sourceNot checked yet

  Not checked in the free run.

• cites [38] Verbakel, 2020, ROC curves for clinical prediction models part 1. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models, J Clin Epidemiol sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Loglikelihood, logloss, and Brier score are strictly proper.

“Loglikelihood, logloss, and Brier score are strictly proper”

• cites [26] BujaNot checked yet

  Not checked in the free run.

• cites [27] Gneiting, 2007, Strictly proper scoring rules, prediction, and estimation, J Am Stat Assoc sourceNot checked yet

  Not checked in the free run.

• cites [71] Van Houwelingen, 2012Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Calibration approaches can be classified into three increasingly stringent levels, labelled as mean, weak, and moderate calibration.

“These approaches can be classified into three increasingly stringent levels, labelled as mean, weak, and moderate calibration. 52”

• cites [52] Van Calster, 2016, A calibration hierarchy for risk models was defined: from utopia to empirical data, J Clin Epidemiol sourceNot checked yet

  Not checked in the free run.

May combine two claims.

AUROC, AUPRC, and pAUROC are semi-proper, because these rank-based measures are invariant to monotonic transformations of probability estimates.

“AUROC, AUPRC, and pAUROC are semi-proper, because these rank-based measures are invariant to monotonic transformations of probability estimates. 48”

• cites [48] Pepe, 2015, The Net Reclassification Index (NRI): a misleading measure of prediction improvement even with independent test data sets, Stat Biosci sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

When maximising the Youden index, sensitivity and specificity are considered equally important; this is rarely the case in medicine.

“When maximising the Youden index, sensitivity and specificity are considered equally important; this is rarely the case in medicine.”

• AssumptionRests on an assumption: The primary aim of most predictive AI models in medical practice is to support subsequent decision making.

Classified as probably not a substantial claim. May combine two claims.

The first four performance domains focus on different aspects of statistical performance, whereas the clinical utility domain focuses on decision-analytical performance.

“The fi rst four performance domains (appendix p 2) focus on different aspects of statistical performance, whereas the clinical utility domain focuses on decision-analytical performance.”

Probably not a substantial claim: may not pass S2 (checkable).

Improper measures might mislead researchers instead of clarifying the performance of a model.

“Improper measures might mislead researchers instead of clarifying the performance of a model.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Classification accuracy, balanced accuracy, Youden index, DOR, kappa, F1, and MCC are improper at clinically relevant decision thresholds; in addition, some measures are hard to interpret.

“These measures are improper at clinically relevant decision thresholds; in addition, some measures are hard to interpret.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Displaying the distribution of risk estimates for each outcome category provides valuable insights into a model's behaviour.

“Displaying the distribution of the risk estimates for each outcome category provides valuable insights into a model's behaviour.”

May combine two claims.

F1 is the only measure violating both characteristics: being improper and not having a clear focus on either statistical or decision-analytical performance.

“Remarkably, F1 is the only measure violating both characteristics.”

May combine two claims.

Net benefit and standardised net benefit are semi-proper; the probability estimates below the threshold can be anything as long as they are below the threshold, and the same goes for probability estimates above the threshold.

“Net benefit and standardised net benefit are semi-proper; the probability estimates below the threshold can be anything as long as they are below the threshold, and the same goes for probability estimates above the threshold.”

• cites [48] Pepe, 2015, The Net Reclassification Index (NRI): a misleading measure of prediction improvement even with independent test data sets, Stat Biosci sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

All discussed overall performance measures have a focus on statistical performance.

“All discussed overall measures have a focus on statistical performance.”

Probably not a substantial claim: may not pass S2 (checkable).

Discrimination and calibration aid the modeller and clinician to understand how a model can be improved.

“Discrimination and calibration aid the modeller and clinician to understand how a model can be improved.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

ROC and PR curves provide limited additional information over AUROC.

“These plots provide limited additional information over AUROC.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Expected cost is semi-proper because it is insensitive to rank-preserving transformations of the probabilities.

“Expected cost is semi-proper because it is insensitive to rank-preserving transformations of the probabilities.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Standardised net benefit equals net benefit divided by prevalence, and its maximum value is 1.

“Standardised net benefit equals net benefit divided by prevalence, and its maximum value is 1.”

• cites [84] Kerr, 2016, Assessing the clinical impact of risk prediction models with decision curves: guidance for correct interpretation and appropriate use, J Clin Oncol sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Calibration intercept and calibration slope are hard to interpret and provide a partial assessment of calibration; during internal validation, calibration slope can be used to gauge overfitting.

“These measures are hard to interpret and provide a partial assessment of calibration; during internal validation, calibration slope can be used to gauge overfitting.”

• cites [66] Kattan, 2018, The index of prediction accuracy: an intuitive measure useful for evaluating risk prediction models, Diagn Progn Res sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A limitation of ROC and PR curves is that the thresholds are not easily visible.

“A limitation of these plots is that the thresholds are not easily visible (figure 1).”

• cites [38] Verbakel, 2020, ROC curves for clinical prediction models part 1. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models, J Clin Epidemiol sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Sensitivity and specificity are largely theoretical as they condition on the predicted outcome.

“However, these measures are largely theoretical as they condition on the predicted outcome.”

May combine two claims.

Setting misclassification costs is not straightforward, and disagreements can occur about what the costs should be.

“Setting misclassification costs is not straightforward, and disagreements can occur about what the costs should be (appendix p 13).”

• cites [85] Vickers, 2019, A simple, step-by-step guide to interpreting decision curve analysis, Diagn Progn Res sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Discrimination slope and MAPE are improper; values can be better for incorrect models than for the correct model.

“These measures are improper; ie, values can be better for incorrect models than for the correct model.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

AUROC quantifies discrimination, which is a key component of statistical model performance.

“This measure quantifies discrimination, which is a key component of statistical model performance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Net benefit sets the decision threshold based on the misclassification costs, following the classic connection between both.

“Net benefit sets the decision threshold based on the misclassification costs, following the classic connection between both.”

• cites [18] Pauker, 1975, Therapeutic decision making: a cost-benefit analysis, N Engl J Med sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Expected cost searches for the decision threshold that minimises cost given the misclassification costs.

“In contrast to net benefit, expected cost searches for the decision threshold that minimises cost given the misclassification costs.”

• cites [86] Margineantu, 2000, Bootstrap methods for the cost-sensitive evaluation of classifiersNot checked yet

  Not checked in the free run.

• cites [87] Ferrer, 2022, Analysis and comparison of classification metrics, arXivNot checked yet

  Not checked in the free run.

• cites [88] Provost, 2001, Robust classification for imprecise environments, Mach Learn sourceNot checked yet

  Not checked in the free run.

• cites [89] Van Calster, 2015, Calibration of risk prediction models: impact on decision-analytic performance, Med Decis Making sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Poor discrimination indicates that other predictors facilitating improved distinction between individuals with and without the event could be selected.

“Poor discrimination indicates that other predictors facilitating improved distinction between individuals with and without the event could be selected.”

May combine two claims.

Miscalibration of the model might be reflected in the decision threshold at which expected cost is minimised, whereas net benefit fixes the threshold such that miscalibration reduces the net benefit value.

“Miscalibration of the model might thus be reflected in the decision threshold at which expected cost is minimised, whereas net benefit fi xes the threshold such that miscalibration reduces the net benefit value.”

• cites [89] Van Calster, 2015, Calibration of risk prediction models: impact on decision-analytic performance, Med Decis Making sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

AUPRC and pAUROC attempt to move beyond a statistical assessment but violate decision-analytical principles.

“These measures attempt to move beyond a statistical assessment but violate decision-analytical principles.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

ECI, ICI, and ECE summarise calibration plots, concealing the nature and direction of miscalibration, and struggle with statistical consistency.

“These measures summarise calibration plots, concealing the nature and direction of miscalibration, and struggle with statistical consistency.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The O:E ratio is interpretable but provides only a partial assessment of calibration; it is often 1 or close to 1 during internal validation.

“This measure is interpretable but provides only a partial assessment of calibration; O:E ratio is often 1 or close to 1 during internal validation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

MCC has no intuitive interpretation.

“Similar to F1, MCC has no intuitive interpretation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

PPV and NPV are highly practical measures because they condition on the classification.

“PPV and NPV are highly practical measures because they condition on the classification.”

Classified as probably not a substantial claim. May combine two claims.

Overall performance measures combine discrimination and calibration performance, making them less informative than separate assessments of discrimination and calibration performance.

“Overall performance measures combine discrimination and calibration performance, making them less informative than separate assessments of discrimination and calibration performance.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

PPV in combination with NPV, or sensitivity in combination with specificity, are improper when used alone.

“These measures are improper when used alone.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

NB or standardised NB (with a decision curve) and EC (with a cost curve) are important measures to quantify to what extent better decisions are made.

“NB or standardised NB (with a decision curve) and EC (with a cost curve) Recommended Important measures to quantify to what extent better decisions are made.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Class imbalance is related to the target population as an epidemiological feature of the data, whereas misclassification costs are clinical concepts that relate to the context of decision making.

“Class imbalance is related to the target population as an epidemiological feature of the data, whereas misclassification costs are clinical concepts that relate to the context of decision making.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The principles also hold for other types of outcomes, such as nominal, ordinal, time-to-event, or competing risk outcomes.

“Nevertheless, the principles also hold for other types of outcomes, such as nominal, ordinal, time-to-event, or competing risk outcomes.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

For well-developed models, an O:E ratio close to 1 is expected.

“we would expect an O:E ratio close to 1 for welldeveloped models.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Competing models can be evaluated using the same core set of measures and visualisations.

“Although competing models can be evaluated using the same core set of measures and visualisations, proper overall measures become more interesting for tasks such as model selection and comparison.”

Probably not a substantial claim: may not pass S4 (based).

Decision curves of NB allow one to show potential clinical utility at various clinically relevant decision thresholds relative to default decisions (and competing models).

“Decision curves of NB allow one to show potential clinical utility at various clinically relevant decision thresholds relative to default decisions (and competing models).”

Classified as probably not a substantial claim.

Adequate sample size is important to evaluate performance with sufficient precision.

“First, adequate sample size is important to evaluate performance with sufficient precision.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Previous recommendations were to include at least 100 to 200 individuals in the smallest outcome category.

“Previous recommendations were to include at least 100 to 200 individuals in the smallest outcome category.”

• cites [52] Van Calster, 2016, A calibration hierarchy for risk models was defined: from utopia to empirical data, J Clin Epidemiol sourceNot checked yet

  Not checked in the free run.

• cites [99] Collins, 2016, Sample size considerations for the external validation of a multivariable prognostic model: a resampling study, Stat Med sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Miscalibration is often not just a problem of the model but a sign that we need to improve our understanding of the various contexts in which the model is validated and used.

“Miscalibration is often not just a problem of the model but a sign that we need to improve our understanding of the various contexts in which the model is validated and used.”

• cites [91] Van Calster, 2023, There is no such thing as a validated prediction model, BMC Med sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Proper overall measures become more interesting for tasks such as model selection and comparison.

“Although competing models can be evaluated using the same core set of measures and visualisations, proper overall measures become more interesting for tasks such as model selection and comparison.”