The DOI matches its Crossref record; 90 of the 90 references it cites were found in a registry or an index; 7 of the 91 sources its claims cite were checked for support.
Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance
Open accessAbstract
Numerous measures have been proposed to illustrate the performance of predictive artificial intelligence (AI) models. Selecting appropriate performance measures is essential for predictive AI models intended for use in medical practice. Poorly performing models are misleading and may lead to wrong clinical decisions that can be detrimental to patients and increase financial costs. In this Viewpoint, we assess the merits of classic and contemporary performance measures when validating predictive AI models for medical practice, focusing on models that estimate probabilities for a binary outcome. We discuss 32 performance measures covering five performance domains (discrimination, calibration, overall performance, classification, and clinical utility) along with corresponding graphical assessments. The first four domains address statistical performance, whereas the fifth domain covers decision-analytical performance. We discuss two key characteristics when selecting a performance measure and explain why these characteristics are important: (1) whether the measure's expected value is optimised when calculated using the correct probabilities (ie, whether it is a proper measure) and (2) whether the measure solely reflects statistical performance or decision-analytical performance by properly accounting for misclassification costs. 17 measures showed both characteristics, 14 showed one, and one (F1 score) showed neither. All classification measures were improper for clinically relevant decision thresholds other than when the threshold was 0·5 or equal to the true prevalence. We illustrate these measures and characteristics using the ADNEX model which predicts the probability of malignancy in women with an ovarian tumour. We recommend the following measures and plots as essential to report: area under the receiver operating characteristic curve, calibration plot, a clinical utility measure such as net benefit with decision curve analysis, and a plot showing probability distributions by outcome category.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This Viewpoint evaluates 32 performance measures for predictive AI models with binary outcomes across five domains and argues that measures should be proper and clearly focused on either statistical or decision-analytical performance. It recommends reporting AUROC, a calibration plot, a clinical utility measure such as net benefit with decision curve analysis, and a plot of probability distributions by outcome category.
Key findings
- Of 32 measures, 17 showed both key characteristics (properness and clear statistical or decision-analytical focus), 14 showed one, and F1 showed neither; all classification measures were improper at clinically relevant decision thresholds other than when the threshold was 0·5 or equal to the true prevalence.
- In the ADNEX external validation case study (n=894; 434 malignant; prevalence 49%), AUROC was 0·911 (95% CI 0·894–0·927); the O:E ratio was 1·228 (1·171–1·288) before recalibration and 1·000 (0·955–1·046) after; net benefit at t=0·1 was 0·443 (0·411–0·475) before and 0·444 (0·411–0·478) after recalibration.
- Recalibration improved all strictly proper measures and left semi-proper measures improved or unchanged, while most improper classification summary measures worsened (e.g., classification accuracy 0·794 to 0·691; F1 0·818 to 0·756; MCC 0·625 to 0·480).
- The recommended core set is AUROC, a smoothed calibration plot, a clinical utility measure such as net benefit with decision curve analysis, and a figure showing probability distributions for each outcome category.
Methods
- Viewpoint assessing classic and contemporary performance measures for predictive AI models that estimate probabilities for a binary outcome; 32 measures were classified into five domains: discrimination, calibration, overall performance, classification, and clinical utility.
- Illustrative case study: external validation of the ADNEX model for predicting malignancy in women with an ovarian tumour, using TransIOTA data from 894 women (434 malignant, prevalence 49%) recruited between 2015 and 2019 in one secondary and five tertiary care centres in four countries.
- All discussed measures were calculated with 95% CIs using the percentile bootstrap method on 1000 bootstrap samples, before and after logistic recalibration; code and risk estimates/outcomes are available in a GitHub repository, while the complete dataset cannot be publicly shared owing to privacy considerations.
Limitations
- Stated: the Viewpoint focused only on performance measures for binary outcomes, did not address counterfactual prediction, could not discuss all measures, and did not directly discuss model comparisons in depth.
- Stated: calibration and performance-measure research is ongoing, including work on strong calibration, quantifying miscalibration, and uncertainty.
- Evident: as a Viewpoint rather than a systematic review, the recommendations are based on selected measures and a single case study (ADNEX external validation), so generalizability to other outcomes and settings is not empirically tested here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
49 Not checked yet · 45 No basis given · 30 Own interpretation · 14 Own result · 3 Supported · 2 Partly supported · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“(3) the absolute value of F1 changes by simply switching the outcome labels (ie, when 1 becomes 0 and 0 becomes 1).”
What the sources say
- Supportedcites [77] Chicco, 2020, The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation, BMC Genomics source
The Background states that F1 varies when the positive and negative classes are swapped. The Methods section also says F1 is not symmetric under class swapping. Because F1 ranges from 0 to 1, a change in F1 is also a change in its absolute value. (LLM, conf=0.94)
- Supportedcites [78] Christen, 2024, A review of the F-measure: its history, properties, criticism, and alternatives, ACM Comput Surv source
Section 3 defines F1 as 2TP/(2TP + FP + FN), and Section 4.1 explicitly notes that it ignores true negatives. Switching which outcome is labeled positive exchanges TP and TN, so the F1 value generally changes even though the classifications do not; it need not change in every possible case. Page numbers were not available in our copy. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“These issues hold for the more general F beta class of measures, of which F1 is a special case.”
What the sources say
- Supportedcites [78] Christen, 2024, A review of the F-measure: its history, properties, criticism, and alternatives, ACM Comput Surv source
Section 3 identifies F1 as the β = 1 case of Fβ. Section 4.3 and Appendix A.1 extend the discussed problem of outcome-dependent precision and recall weights to the general Fβ measure. The sentence does not specify which issues “these issues” refers to, but the copy supports this generalization for that issue. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Dedicated measures, such as the widely used but improper net reclassification improvement, are available for evaluating competing models.”
What the sources say
- Supportedcites [48] Pepe, 2015, The Net Reclassification Index (NRI): a misleading measure of prediction improvement even with independent test data sets, Stat Biosci source
Section 5.1, ‘NRI is not a Proper Measure of Performance Improvement,’ explicitly states that the NRI is not proper and explains that it can indicate improvement for an incorrect risk function. The paper also describes the NRI as a popular measure for evaluating prediction improvement, matching its role in the citing sentence. (LLM, conf=0.94)
- Partly supportedcites [118] Pencina, 2008, Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond, Stat Med sourceExperimental — accuracy not yet measured
The indexed copy’s section 3.2.1 describes NRI as a measure for comparing models and calls it popular. It cautions that NRI can inflate apparent performance in some circumstances, but does not characterize NRI as categorically improper. No page number is available for the passage. (LLM, conf=0.82)
- Not checked yetcites [119] Leening, 2014, Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician’s guide, Ann Intern Med source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“All classification measures were improper for clinically relevant decision thresholds other than when the threshold was 0 ⋅ 5 or equal to the true prevalence.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The plots lie mostly above the diagonal”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ADNEX had an O:E ratio of 1 ⋅ 23 (95% CI 1 ⋅ 17-1 ⋅ 29), indicating that 23% more events were observed than expected based on the model (table 1).”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our case study, the ADNEX model had a calibration slope of 0 ⋅ 93 (95% CI 0 ⋅ 83-1 ⋅ 05)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The improper summary measures for classification (except diagnostic odds ratio) worsened remarkably.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 3 shows violin plots for the ADNEX model, indicating that patients with benign tumour mostly had very low estimated probabilities of malignancy.”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The improper measures for overall performance improved.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The (standardised) net benefit was better for ADNEX than for the reference strategies across all reasonable decision thresholds (figure 4A-B).”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The range of reasonable decision thresholds is 0.05 to 0.40 (normalised cost of false negatives between 0.60 and 0.95).
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For a normalised cost of a false negative of 0 ⋅ 9, expected cost was minimised to 0 ⋅ 35 at t =0 ⋅ 06.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Patients with malignancy mostly had moderate-to-high estimated probabilities, with less peaked distribution.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All strictly proper measures improved after recalibration.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The net benefit of the model at t =0 ⋅ 1 was 0 ⋅ 44.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Semi-proper measures either improved or remained unchanged.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The calibration plot is closer to the diagonal after recalibration in the validation dataset.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The expected cost curve gives the same impression (figure 4C).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The range of reasonable decision thresholds is 0.05 to 0.40 (normalised cost of false negatives between 0.60 and 0.95).
None of the sources cited for this claim have been checked yet.
In the paper
“Unfortunately, calibration measures are still under-reported.”
What the sources say
- Not checked yetcites [92] Wynants, 2020, Prediction models for diagnosis and prognosis of COVID-19: systematic review and critical appraisal, BMJ
- Not checked yetcites [93] Andaur Navarro, 2021, Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review, BMJ
- Not checked yetcites [94] Dhiman, 2022, Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review, BMC Med Res Methodol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The ADNEX model, developed by the International Ovarian Tumor Analysis (IOTA) consortium (authors BVC and DT are part of this consortium), preoperatively estimates the probability of malignancy in women with an ovarian tumour who are scheduled for surgery. 31”
What the sources say
- Not checked yetcites [31] Van Calster, 2014, Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study, BMJ source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Logistic recalibration essentially applies a linear transformation to the linear predictor.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“scaled Brier and R-squared measures are asymptotically strictly proper (ie, strictly proper when sample size is high, for instance, above 100)”
What the sources say
- Not checked yetcites [26] Buja
- Not checked yetcites [27] Gneiting, 2007, Strictly proper scoring rules, prediction, and estimation, J Am Stat Assoc source
- Not checked yetcites [71] Van Houwelingen, 2012
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting that the spread of the probabilities was adequate.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, we accept up to nine false positives (ie, performing advanced surgery in up to nine patients with benign tumour) per true positive.”
plaindirectn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: Using a decision threshold of 0.1 for probability of malignancy assumes that the medical benefit of advanced surgery on a malignant tumour is nine times greater than the harm of unnecessary advanced surgery in individuals with a benign tumour.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Grouped plots cannot be used to comprehensively address moderate calibration because individuals with very different estimated probabilities might still end up in the same group.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“with a focus on statistical performance (second key characteristic).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“During internal validation, a calibration slope less than 1 can indicate potential overfitting.”
What the sources say
- Not checked yetcites [54] Van Calster, 2019, Calibration: the Achilles heel of predictive analytics, BMC Med source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For AUPRC, the PR curve does not directly consider true negatives.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A probable reason is that fi ve of six participating centres in the validation study were tertiary care centres, resulting in a high prevalence of malignancy (49%).”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Suggesting advanced surgery to patients who actually have a 10% risk of malignancy based on the ADNEX predictors implies performing advanced surgery in ten individuals per true positive (ie, performing advanced surgery in a patient with a malignant tumour). 18”
What the sources say
- Not checked yetcites [18] Pauker, 1975, Therapeutic decision making: a cost-benefit analysis, N Engl J Med source
plaindirectn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: Using a decision threshold of 0.1 for probability of malignancy assumes that the medical benefit of advanced surgery on a malignant tumour is nine times greater than the harm of unnecessary advanced surgery in individuals with a benign tumour.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Consequently, there is no decision-analytical basis for this approach.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting that the probabilities were underestimations across the whole range.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“All discussed calibration measures are semi-proper, 52”
What the sources say
- Not checked yetcites [52] Van Calster, 2016, A calibration hierarchy for risk models was defined: from utopia to empirical data, J Clin Epidemiol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In any given dataset, particularly when sample size is low, the correct model can be outperformed by an incorrect model due to random variation.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Similar to statistical tests such as the Hosmer-Lemeshow test, the proposed summary measures cannot inform on the direction of miscalibration.”
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“There are no grounds to label AUROC as misleading or overoptimistic. 36,49”
What the sources say
- Not checked yetcites [36] Berrar, 2012, Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them), Brief Bioinform source
- Not checked yetcites [49] McDermott, 2024, A closer look at AUROC and AUPRC under class imbalance, Adv Neural Inf Process Syst
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike AUROC, AUPRC and pAUROC do not have a clear focus (second key characteristic).”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The worsening of most improper performance measures after recalibration illustrates the importance of the properness concept.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, owing to the rankpreserving updating method, recalibration cannot improve discrimination measures such as AUROC as they are based on ranks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“The most used measure for clinical utility in prediction studies in health care is net benefit.”
What the sources say
- Not checked yetcites [21] Vickers, 2006, Decision curve analysis: a novel method for evaluating prediction models, Med Decis Making source
- Not checked yetcites [82] Vickers, 2016, Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests, BMJ
- Not checked yetcites [83] Kerr, 2019, Assessing the clinical impact of risk models for opting out of treatment, Med Decis Making source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Some classification measures (balanced accuracy, Youden, and F1) are semi-proper when t =0 ⋅ 5 (classification accuracy) or when t equals the true prevalence”
What the sources say
- Not checked yetcites [26] Buja
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“(1) F1 ignores true negatives”
What the sources say
- Not checked yetcites [77] Chicco, 2020, The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation, BMC Genomics source
- Not checked yetcites [78] Christen, 2024, A review of the F-measure: its history, properties, criticism, and alternatives, ACM Comput Surv source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“F1 is the only summary measure without a clear focus on statistical performance, as it conflates classification with clinical utility.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“At a given relevant decision threshold t , all classification measures are improper (Appendix pp 3-6).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The maximum value of net benefit equals the prevalence.”
What the sources say
- Not checked yetcites [84] Kerr, 2016, Assessing the clinical impact of risk prediction models with decision curves: guidance for correct interpretation and appropriate use, J Clin Oncol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“F1 and MCC were introduced to address the challenges caused by class imbalance, where classification accuracy can be inflated by classifying all patients as low risk when the event is rare.”
What the sources say
- Not checked yetcites [76] Van Rijsbergen, 1979
- Not checked yetcites [77] Chicco, 2020, The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation, BMC Genomics source
- Not checked yetcites [78] Christen, 2024, A review of the F-measure: its history, properties, criticism, and alternatives, ACM Comput Surv source
- Not checked yetcites [79] Baldi, 2000, Assessing the accuracy of prediction algorithms for classification: an overview, Bioinformatics source
- Not checked yetcites [80] Chicco, 2021, The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation, BioData Min source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“(2) F1 has no intuitive interpretation”
What the sources say
- Not checked yetcites [77] Chicco, 2020, The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation, BMC Genomics source
- Not checked yetcites [78] Christen, 2024, A review of the F-measure: its history, properties, criticism, and alternatives, ACM Comput Surv source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Meta-analysis and meta-regression methods can be used to quantify and understand heterogeneity in performance across external validation studies.”
What the sources say
- Not checked yetcites [102] van Leeuwen, 2025, Instability of the AUROC of clinical prediction models, Stat Med source
- Not checked yetcites [105] Debray, 2019, A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes, Stat Methods Med Res source
- Not checked yetcites [106] Barreñada, 2024, ADNEX risk prediction model for diagnosis of ovarian cancer: systematic review and meta-analysis of external validation studies, BMJ Med source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“More specific sample size calculations are now available for regression-based models.”
What the sources say
- Not checked yetcites [100] Riley, 2021, Minimum sample size for external validation of a clinical prediction model with a binary outcome, Stat Med source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Naive comparison of models validated using different external datasets, reflecting different populations from different settings, can lead to wrong conclusions.”
What the sources say
- Not checked yetcites [36] Berrar, 2012, Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them), Brief Bioinform source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We argue that class imbalance is not as problematic as often claimed.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Miscalibration can compromise predictive AI application by leading to systematic overtreatment or undertreatment.”
What the sources say
- Not checked yetcites [89] Van Calster, 2015, Calibration of risk prediction models: impact on decision-analytic performance, Med Decis Making source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Often, more data are needed when comparing calibration between models.”
What the sources say
- Not checked yetcites [101] Peek, 2007, External validation of prognostic models for critically ill patients required substantial sample sizes, J Clin Epidemiol source
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Calibration is more important for external validation, when models are evaluated in different contexts and populations.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Second, heterogeneity in model performance should be expected based on differences in populations and measurement procedures between locations, settings, or time periods.”
What the sources say
- Not checked yetcites [91] Van Calster, 2023, There is no such thing as a validated prediction model, BMC Med source
- Not checked yetcites [102] van Leeuwen, 2025, Instability of the AUROC of clinical prediction models, Stat Med source
- Not checked yetcites [103] Youssef, 2023, External validation of AI models in health should be replaced with recurring local validation, Nat Med source
- Not checked yetcites [104] Jenkins, 2021, Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems?, Diagn Progn Res source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Misclassification costs are informed by the nature and effect of the medical intervention at hand (eg, the decision to perform surgery or not).”
What the sources say
- Not checked yetcites [21] Vickers, 2006, Decision curve analysis: a novel method for evaluating prediction models, Med Decis Making source
- Not checked yetcites [82] Vickers, 2016, Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests, BMJ
- Not checked yetcites [85] Vickers, 2019, A simple, step-by-step guide to interpreting decision curve analysis, Diagn Progn Res source
- Not checked yetcites [98] Hand, 2018, A note on using the F-measure for evaluating record linkage algorithms, Stat Comput source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The extent of class imbalance is not mathematically proportional to the extent of imbalance in misclassification costs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 47
- Probably not a substantial claim. 62
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Estimated probabilities with too much spread are on average too close to 0 and 1 (slope < 1), whereas estimated probabilities with too little spread are on average too close to the prevalence (slope > 1).”
What the sources say
- Partly supportedcites [52] Van Calster, 2016, A calibration hierarchy for risk models was defined: from utopia to empirical data, J Clin Epidemiol sourceExperimental — accuracy not yet measured
Section 2.1 (page unavailable) says overfitting produces predictions that are too extreme: high risks are overestimated and low risks underestimated. It also identifies the calibration slope as an indicator of overfitting or underfitting, but our copy does not explicitly state that underfitted probabilities are too close to the prevalence; the numerical slope thresholds are missing from its rendered equations. (LLM, conf=0.78)
- Not checked yetcites [53] Stevens, 2020, Validation of clinical prediction models: what does the “calibration slope” really measure?, J Clin Epidemiol source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“There were 434 women with a malignant tumour (prevalence 49%). 32”
What the sources say
- Supportedcites [32] Landolfo, 2024, Comparison of the ADNEX and ROMA risk prediction models for the diagnosis of ovarian cancer: a multicentre external validation in patients who underwent surgery, Br J Cancer source
The Patient and tumour characteristics section states that the primary analysis included 894 patients, of whom 434 (49%) had a malignant tumour. This directly matches the cited claim. (LLM, conf=0.95)
plainn = 434- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The ADNEX model had an AUROC of 0 ⋅ 91 (95% CI 0 ⋅ 89-0 ⋅ 93) and an AUPRC of 0 ⋅ 89 (95% CI 0 ⋅ 86-0 ⋅ 91).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The calibration intercept of the ADNEX model was 0 ⋅ 81 (95% CI 0 ⋅ 62-1 ⋅ 01), suggesting underestimation of probabilities on average (intercept > 0).”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Ignoring sensitivity values below 0 ⋅ 8, pAUROC was 0 ⋅ 14 (95% CI 0 ⋅ 13-0 ⋅ 15).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“A probable reason is that fi ve of six participating centres in the validation study were tertiary care centres, resulting in a high prevalence of malignancy (49%).”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“17 measures showed both characteristics, 14 showed one, and one (F1 score) showed neither.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some partial classification measures improved (sensitivity and NPV), whereas others worsened (specificity and PPV).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At a threshold of 10%, the ADNEX model showed a classification accuracy of 0 ⋅ 79 (95% CI 0 ⋅ 77-0 ⋅ 82), F1 score of 0 ⋅ 82 (0 ⋅ 79-0 ⋅ 84), and MCC of 0 ⋅ 63 (0 ⋅ 58-0 ⋅ 67) (table 1).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“At the commonly recommended threshold of 10% in our case study, 23 ADNEX classified 578 patients as high risk, of whom 414 had a malignant tumour (true positive) and 164 had a benign tumour (false positive). The remaining 316 patients were classified as low risk, of whom 296 had a benign tumour (true negative) and 20 had a malignant tumour (false negative).”
What the sources say
- Not checked yetcites [23] Timmerman, 2021, ESGO/ISUOG/IOTA/ESGE consensus statement on preoperative diagnosis of ovarian tumors, Ultrasound Obstet Gynecol source
plainn = 578- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“For example, calibration is an active area of research focusing on aspects such as strong calibration, quantifying the degree of miscalibration, and uncertainty.”
What the sources say
- Not checked yetcites [62] Arrieta-Ibarra, 2022, Metrics of calibration for probabilistic predictions, J Mach Learn Res
- Not checked yetcites [113] Perez-Lebel, 2022, Beyond calibration: estimating the grouping loss of modern neural networks, arXiv
- Not checked yetcites [114] Kompa, 2021, Second opinion needed: communicating uncertainty in medical machine learning, NPJ Digit Med source
- Not checked yetcites [115] Riley, 2023, Stability of clinical prediction models developed using statistical or machine learning methods, Biom J source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“If misclassification costs influence a performance measure in an implicit or ad hoc way, the measure neither assesses statistical performance nor adequately evaluates the quality of decisions for clinical practice.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Discrimination reflects relative performance; ie, it does not matter how high or low the estimated probabilities are, only whether they allow to discriminate between individuals with versus without the event.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Classification performance is influenced by discrimination and calibration performance.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Using statistical arguments to set a decision threshold is inconsistent with decision theory and detached from practical use by clinicians.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
- Rests on an assumption: The primary aim of most predictive AI models in medical practice is to support subsequent decision making.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Statistical performance measures are essential for model evaluation but cannot be used to fi nd out whether a model should be used in practice: it is not appropriate to cite, for example, good discrimination and calibration or conclude that a model can be used to aid decisions about ovarian surgery.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The fi fth domain, clinical utility, goes one step further by explicitly incorporating misclassification costs when evaluating classifications of individuals into low-risk and high-risk groups.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Among these domains, the fi rst three evaluate performance based on probability estimates (Appendix p 2).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When an incorrect model can have a better expected value than the correct model, the measure is termed improper and cannot be trusted.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The importance of properness is that a proper measure cannot be fooled: in expectation, the correct model cannot be outperformed by an incorrect one.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The fourth and fi fth performance domains require a threshold on the estimated risk of the event to classify individuals into two mutually exclusive groups: low-risk (estimated risk below the threshold) and high-risk (estimated risk equal to or above the threshold) groups.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Measures that do not possess the fi rst characteristic cannot be trusted, whereas measures not possessing the second are equivocal.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“poorly performing models might lead to wrong clinical decisions that can be detrimental to patients and increase fi nancial costs. 5”
What the sources say
- Not checked yetcites [5] Kamran, 2024, Evaluation of sepsis prediction models before onset of treatment, NEJM AI source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Calibration reflects absolute performance by evaluating whether estimates are too high or too low.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Models can therefore have good discrimination but poor calibration, and vice versa.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The definition of discrimination implies that discrimination measures should rely only on the ranks of the estimated probabilities in the dataset. 36”
What the sources say
- Not checked yetcites [36] Berrar, 2012, Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them), Brief Bioinform source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Research on the quantification of a fourth level, strong calibration, is ongoing.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“A decision threshold of 0 ⋅ 1 (10%) for the probability of malignancy is often recommended. 23”
What the sources say
- Not checked yetcites [23] Timmerman, 2021, ESGO/ISUOG/IOTA/ESGE consensus statement on preoperative diagnosis of ovarian tumors, Ultrasound Obstet Gynecol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: The primary aim of most predictive AI models in medical practice is to support subsequent decision making.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Discrimination measures are not meant to reflect differential misclassification costs, and class imbalance should not be conflated with misclassification costs or medical relevance.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“ADNEX was developed on the data from 5909 individuals recruited between 1999 and 2012 in 24 secondary and tertiary care centres from 10 countries (Italy, Belgium, Sweden, Czech Republic, Poland, France, United Kingdom, China, Spain, and Canada).”
What the sources say
- Not checked yetcites [31] Van Calster, 2014, Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study, BMJ source
plainn = 5,909- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The model can inform decisions regarding the type of surgery (advanced vs conservative) for patients examined at an oncology centre or regarding referral to an oncology centre for patients examined elsewhere. 32”
What the sources say
- Not checked yetcites [32] Landolfo, 2024, Comparison of the ADNEX and ROMA risk prediction models for the diagnosis of ovarian cancer: a multicentre external validation in patients who underwent surgery, Br J Cancer source
plaindirectn = 894- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This method is thus a rank-preserving method, and the ranking of patients based on the estimated probability of malignancy remains the same before and after updating.”
plainn = 894- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The weight of these consequences might vary by the nature and effects of the intervention, the health-care system, or by clinician and patient.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: The primary aim of most predictive AI models in medical practice is to support subsequent decision making.
In the paper
“Further, ECE, ECI, and ICI depend on the grouping or smoothing methods used and have issues with statistical consistency.”
What the sources say
- Not checked yetcites [62] Arrieta-Ibarra, 2022, Metrics of calibration for probabilistic predictions, J Mach Learn Res
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it is not appropriate to cite, for example, good discrimination and calibration or conclude that a model can be used to aid decisions about ovarian surgery.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Often, however, a threshold is chosen by optimising a statistical measure such as the Youden index (sensitivity + specificity -1).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: The primary aim of most predictive AI models in medical practice is to support subsequent decision making.
None of the sources cited for this claim have been checked yet.
In the paper
“discrimination slope and mean absolute prediction error are improper.”
What the sources say
- Not checked yetcites [26] Buja
- Not checked yetcites [27] Gneiting, 2007, Strictly proper scoring rules, prediction, and estimation, J Am Stat Assoc source
- Not checked yetcites [71] Van Houwelingen, 2012
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Although true negatives might be highly irrelevant for selected non-medical applications, such errors are generally important in medical applications.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Nevertheless, although discriminatory ability is essential for predictive AI, AUROC alone cannot be used to identify a model as good or useful. 51”
What the sources say
- Not checked yetcites [51] de Hond, 2022, Interpreting area under the receiver operating characteristic curve, Lancet Digit Health source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“This method is similar to Platt scaling, a well known method in machine learning to improve the calibration of predictions. 34,35”
What the sources say
- Not checked yetcites [34] Ojeda, 2023, Calibrating machine learning approaches for probability estimation: a comprehensive comparison, Stat Med source
- Not checked yetcites [35] Platt, 1999, Probabilistic outputs for support vector machines and comparison to regularized likelihood methods
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The TransIOTA study externally validated ADNEX for its ability to distinguish benign from malignant tumours, using data from 894 women recruited between 2015 and 2019 in one secondary and five tertiary care centres in four countries (Belgium, Italy, Czech Republic, and United Kingdom).”
What the sources say
- Not checked yetcites [32] Landolfo, 2024, Comparison of the ADNEX and ROMA risk prediction models for the diagnosis of ovarian cancer: a multicentre external validation in patients who underwent surgery, Br J Cancer source
plaincorrelationaln = 894- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In biomedical applications, the consequences of a false negative (for instance, not referring a woman with ovarian malignancy for advanced surgery) are almost always different from the consequences of a false positive (referring a woman with a benign tumour for advanced surgery).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AUPRC and pAUROC mix statistical performance with aspects of clinical utility without following decision-analytical principles.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“For binary outcomes, the C-statistic is equal to area under the receiver operating characteristic curve (AUROC). 37,38”
What the sources say
- Not checked yetcites [37] Bradley, 1997, The use of the area under the ROC curve in the evaluation of machine learning algorithms, Pattern Recognit source
- Not checked yetcites [38] Verbakel, 2020, ROC curves for clinical prediction models part 1. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models, J Clin Epidemiol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Loglikelihood, logloss, and Brier score are strictly proper”
What the sources say
- Not checked yetcites [26] Buja
- Not checked yetcites [27] Gneiting, 2007, Strictly proper scoring rules, prediction, and estimation, J Am Stat Assoc source
- Not checked yetcites [71] Van Houwelingen, 2012
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“These approaches can be classified into three increasingly stringent levels, labelled as mean, weak, and moderate calibration. 52”
What the sources say
- Not checked yetcites [52] Van Calster, 2016, A calibration hierarchy for risk models was defined: from utopia to empirical data, J Clin Epidemiol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“AUROC, AUPRC, and pAUROC are semi-proper, because these rank-based measures are invariant to monotonic transformations of probability estimates. 48”
What the sources say
- Not checked yetcites [48] Pepe, 2015, The Net Reclassification Index (NRI): a misleading measure of prediction improvement even with independent test data sets, Stat Biosci source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When maximising the Youden index, sensitivity and specificity are considered equally important; this is rarely the case in medicine.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: The primary aim of most predictive AI models in medical practice is to support subsequent decision making.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The fi rst four performance domains (appendix p 2) focus on different aspects of statistical performance, whereas the clinical utility domain focuses on decision-analytical performance.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Improper measures might mislead researchers instead of clarifying the performance of a model.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These measures are improper at clinically relevant decision thresholds; in addition, some measures are hard to interpret.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Displaying the distribution of the risk estimates for each outcome category provides valuable insights into a model's behaviour.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Remarkably, F1 is the only measure violating both characteristics.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Net benefit and standardised net benefit are semi-proper; the probability estimates below the threshold can be anything as long as they are below the threshold, and the same goes for probability estimates above the threshold.”
What the sources say
- Not checked yetcites [48] Pepe, 2015, The Net Reclassification Index (NRI): a misleading measure of prediction improvement even with independent test data sets, Stat Biosci source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“All discussed overall measures have a focus on statistical performance.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Discrimination and calibration aid the modeller and clinician to understand how a model can be improved.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These plots provide limited additional information over AUROC.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Expected cost is semi-proper because it is insensitive to rank-preserving transformations of the probabilities.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Standardised net benefit equals net benefit divided by prevalence, and its maximum value is 1.”
What the sources say
- Not checked yetcites [84] Kerr, 2016, Assessing the clinical impact of risk prediction models with decision curves: guidance for correct interpretation and appropriate use, J Clin Oncol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“These measures are hard to interpret and provide a partial assessment of calibration; during internal validation, calibration slope can be used to gauge overfitting.”
What the sources say
- Not checked yetcites [66] Kattan, 2018, The index of prediction accuracy: an intuitive measure useful for evaluating risk prediction models, Diagn Progn Res source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“A limitation of these plots is that the thresholds are not easily visible (figure 1).”
What the sources say
- Not checked yetcites [38] Verbakel, 2020, ROC curves for clinical prediction models part 1. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models, J Clin Epidemiol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, these measures are largely theoretical as they condition on the predicted outcome.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Setting misclassification costs is not straightforward, and disagreements can occur about what the costs should be (appendix p 13).”
What the sources say
- Not checked yetcites [85] Vickers, 2019, A simple, step-by-step guide to interpreting decision curve analysis, Diagn Progn Res source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These measures are improper; ie, values can be better for incorrect models than for the correct model.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This measure quantifies discrimination, which is a key component of statistical model performance.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Net benefit sets the decision threshold based on the misclassification costs, following the classic connection between both.”
What the sources say
- Not checked yetcites [18] Pauker, 1975, Therapeutic decision making: a cost-benefit analysis, N Engl J Med source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“In contrast to net benefit, expected cost searches for the decision threshold that minimises cost given the misclassification costs.”
What the sources say
- Not checked yetcites [86] Margineantu, 2000, Bootstrap methods for the cost-sensitive evaluation of classifiers
- Not checked yetcites [87] Ferrer, 2022, Analysis and comparison of classification metrics, arXiv
- Not checked yetcites [88] Provost, 2001, Robust classification for imprecise environments, Mach Learn source
- Not checked yetcites [89] Van Calster, 2015, Calibration of risk prediction models: impact on decision-analytic performance, Med Decis Making source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Poor discrimination indicates that other predictors facilitating improved distinction between individuals with and without the event could be selected.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Miscalibration of the model might thus be reflected in the decision threshold at which expected cost is minimised, whereas net benefit fi xes the threshold such that miscalibration reduces the net benefit value.”
What the sources say
- Not checked yetcites [89] Van Calster, 2015, Calibration of risk prediction models: impact on decision-analytic performance, Med Decis Making source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These measures attempt to move beyond a statistical assessment but violate decision-analytical principles.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These measures summarise calibration plots, concealing the nature and direction of miscalibration, and struggle with statistical consistency.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This measure is interpretable but provides only a partial assessment of calibration; O:E ratio is often 1 or close to 1 during internal validation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Similar to F1, MCC has no intuitive interpretation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“PPV and NPV are highly practical measures because they condition on the classification.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall performance measures combine discrimination and calibration performance, making them less informative than separate assessments of discrimination and calibration performance.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These measures are improper when used alone.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“NB or standardised NB (with a decision curve) and EC (with a cost curve) Recommended Important measures to quantify to what extent better decisions are made.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Class imbalance is related to the target population as an epidemiological feature of the data, whereas misclassification costs are clinical concepts that relate to the context of decision making.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nevertheless, the principles also hold for other types of outcomes, such as nominal, ordinal, time-to-event, or competing risk outcomes.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we would expect an O:E ratio close to 1 for welldeveloped models.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although competing models can be evaluated using the same core set of measures and visualisations, proper overall measures become more interesting for tasks such as model selection and comparison.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Decision curves of NB allow one to show potential clinical utility at various clinically relevant decision thresholds relative to default decisions (and competing models).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, adequate sample size is important to evaluate performance with sufficient precision.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Previous recommendations were to include at least 100 to 200 individuals in the smallest outcome category.”
What the sources say
- Not checked yetcites [52] Van Calster, 2016, A calibration hierarchy for risk models was defined: from utopia to empirical data, J Clin Epidemiol source
- Not checked yetcites [99] Collins, 2016, Sample size considerations for the external validation of a multivariable prognostic model: a resampling study, Stat Med source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Miscalibration is often not just a problem of the model but a sign that we need to improve our understanding of the various contexts in which the model is validated and used.”
What the sources say
- Not checked yetcites [91] Van Calster, 2023, There is no such thing as a validated prediction model, BMC Med source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although competing models can be evaluated using the same core set of measures and visualisations, proper overall measures become more interesting for tasks such as model selection and comparison.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The authors state this as an assumption.
Using a decision threshold of 0.1 for probability of malignancy assumes that the medical benefit of advanced surgery on a malignant tumour is nine times greater than the harm of unnecessary advanced surgery in individuals with a benign tumour.
“Hence, using this threshold assumes that the medical benefit of advanced surgery on a malignant tumour is nine times greater than the harm of unnecessary advanced surgery in individuals with a benign tumour. 18”
The paper relies on this without support.
The primary aim of most predictive AI models in medical practice is to support subsequent decision making.
“The primary aim of most predictive AI models in medical practice is to support subsequent decision making.”
The authors state this as an assumption.
The range of reasonable decision thresholds is 0.05 to 0.40 (normalised cost of false negatives between 0.60 and 0.95).
“We further assume that the range of reasonable thresholds is 0 ⋅ 05 to 0 ⋅ 40 (normalised cost of false negatives between 0 ⋅ 60 and 0 ⋅ 95).”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Performance measures · p. 4
“| Logloss or cross entropy | ++ | + | 370 (334 to 407) | 377 (307 to 368) |”
§ Performance measures · p. 4
“| Loglikelihood | ++ | + | -370 (-407 to -334) | -337 (-368 to -307) |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Discrimination · p. 5
“an AUPRC of 0 ⋅ 89 (95% CI 0 ⋅ 86-0 ⋅ 91)”
§ Performance measures · p. 4
“| AUPRC or AP | + | - | 0 ⋅ 895 (0 ⋅ 862 to 0 ⋅ 921) | 0 ⋅ 895 (0 ⋅ 862 to 0 ⋅ 921) |”
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
The use of the area under the ROC curve in the evaluation of machine learning algorithms
Bradley, A. P. (1997) · Pattern Recognition · doi:10.1016/s0031-3203(96)00142-2 · Cited by 7,217
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Self-citationChicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143
A note on a general definition of the coefficient of determination
NAGELKERKE, N. J. D. (1991) · Biometrika · doi:10.1093/biomet/78.3.691 · Cited by 5,927
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Self-citationMoons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citation
Research works printed but not linked: 12
- Hernández-Orallo, 2012, A unified view of performance metrics: translating threshold choice of into expected classification loss, J Mach Learn Res, 13, 2813
- Riley, 2024, Evaluation of clinical prediction models (part 2): how to undertake an external validation study, BMJ, 384
- McDermott, 2024, A closer look at AUROC and AUPRC under class imbalance, Adv Neural Inf Process Syst, 37, 44102
- Błasiok, 2023, Smooth ECE: principled reliability diagrams via kernel smoothing, arXiv
- Arrieta-Ibarra, 2022, Metrics of calibration for probabilistic predictions, J Mach Learn Res, 23, 1
- Vickers, 2016, Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests, BMJ, 352
- Ferrer, 2022, Analysis and comparison of classification metrics, arXiv
- Wynants, 2020, Prediction models for diagnosis and prognosis of COVID-19: systematic review and critical appraisal, BMJ, 369
- Andaur Navarro, 2021, Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review, BMJ, 375
- Debray, 2023, Transparent reporting of multivariable prediction models developed or validated using clustered data (TRIPOD-Cluster): explanation and elaboration, BMJ, 380
- Perez-Lebel, 2022, Beyond calibration: estimating the grouping loss of modern neural networks, arXiv
- Gupta, 2020, Evaluation of machine learning algorithms for predicting readmission after acute myocardial infarction using routinely collected clinical data, Can J Cardiol, 36, 878
Cited by
Sorted by citations
Applications of Artificial Intelligence in Fisheries: From Data to Decisions
Haque, S. A., & Jufaili, S. M. A. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10010019 · Cited by 18
Yang, H. S., Çubukçu, H. C., Ben, F. D., Duan, X., Feng, J., Frans, G., Gruson, D., King, M. C., Nieto, J., Wang, B., et al. (2026) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2025-1322 · Cited by 17
“…performance, classi fi cation, and clinical utility, with area under the receiver operating characteristic curve (AUROC), calibration plots, clinical utility measures (e.g., net bene fi t with decision curve analysis), and outcome-strati fi ed probability distribution as essential elements [30].”
Prospective evaluation of balance and mobility tests as part of the World Falls Guidelines algorithm
Saunders, S., Speechley, M. R., Griffith, L. E., Kuspinar, A., Sibley, K. M., Raina, P., D’Amore, C., Richardson, J. A., Noble, T., & Beauchamp, M. K. (2026) · Age and Ageing · doi:10.1093/ageing/afag028 · Cited by 6
Benitez-Aurioles, J., Wynants, L., Peek, N., et al. (2026) · Diagnostic and Prognostic Research · doi:10.1186/s41512-026-00224-z · Cited by 3
Self-citationPerformance of a Sepsis Prediction Model Across Different Sepsis Definitions
Dutta, S., McMurry, R., Tasi, M. C., Dunham, L. N., McEvoy, D. S., Stump, T., Filbin, M. R., & Rhee, C. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.5599 · Cited by 6
Metadata
- Authors
- Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., Timmerman, D., McLernon, D. J., van Smeden, M., & Steyerberg, E. W.
- Affiliations
- Memorial Sloan Kettering Cancer Center (US)
- Maastricht University (NL)
- University of Washington (US)
- KU Leuven (BE)
- Institut national de recherche en sciences et technologies du numérique (FR)
- University of California San Diego (US)
- Utrecht University (NL)
- University of Aberdeen (GB)
- Corresponding author
- Steyerberg, Ewout W
- ORCID
- Venue
- The Lancet Digital Health
- Publisher
- Elsevier BV
- Published
- 2025-12
- Volume
- 7
- Issue
- 12
- Pages
- 100916
- ISSN
- 2589-7500 (ISSN-L)
- OpenAlex
- W4417296978
- Semantic Scholar
- 3475c68ea3f9f7049202f1d964074c0ab5761021
- Cited by
- 112
- References
- 90
- Citations by year
- 2026: 111 · 2025: 1
- FWCI
- 6.62
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- The Lancet Digital Health (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- Leiden Repository (Leiden University) (submittedVersion, cc-by)
- Lirias (submittedVersion, cc-by)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, cc-by)
- HAL (submittedVersion, cc-by)
- Aberdeen University Research Archive (Aberdeen University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Leiden Repository (Leiden University) (submittedVersion)
- Funders
- National Institutes of Health National Cancer Institute (P30CA008748)
- Engineering and Physical Sciences Research Council (EP/Y018516/1)
- National Center for Advancing Translational Sciences (C24/15/037, UL1TR003142, 12F3114N, G0B4716N, G049312N)
- Foundation for the National Institutes of Health
- Cancer Research UK (MR/V038168/1, C49297/A27294)
- Research Foundation Flanders (G097322N)
- National Institute of Diabetes and Digestive and Kidney Diseases (R42AI177108, ANR-22-PESN-0003, R01 DK093770, R01DK133226, R01EB030492)
- KU Leuven Research Council
- National Institute for Health and Care Research
- National Institutes of Health
- KU Leuven (C24M/20/064)
- National Library of Medicine
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (9150172310023)
- Fight against Cancer
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- performance indicators, clinical decision support, binary outcome prediction, proper scoring rules, calibration, discrimination (statistics), clinical utility, net benefit, decision curve analysis, misclassification cost, ADNEX model, model validation
- MeSH terms
- Clinical Decision-Making, Artificial Intelligence, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
