Validation of clinical prediction models: what does the “calibration slope” really measure?
Open accessAbstract
BACKGROUND AND OBJECTIVES: Definitions of calibration, an aspect of model validation, have evolved over time. We examine use and interpretation of the statistic currently referred to as the calibration slope. METHODS: The history of the term "calibration slope", and usage in papers published in 2016 and 2017, were reviewed. The behaviour of the slope in illustrative hypothetical examples and in two examples in the clinical literature was demonstrated. RESULTS: The paper in which the statistic was proposed described it as a measure of "spread" and did not use the term "calibration". In illustrative examples, slope of 1 can be associated with good or bad calibration, and this holds true across different definitions of calibration. In data extracted from a previous study, the slope was correlated with discrimination, not overall calibration. Many authors of recent papers interpret the slope as a measure of calibration; a minority interpret it as a measure of discrimination or do not explicitly categorise it as either. Seventeen of thirty-three papers used the slope as the sole measure of calibration. CONCLUSION: Misunderstanding about this statistic has led to many papers in which it is the sole measure of calibration, which should be discouraged.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Estimated probabilities with too much spread are on average too close to 0 and 1 (calibration slope < 1), whereas estimated probabilities with too little spread are on average too close to the prevalence (calibration slope > 1).
“Estimated probabilities with too much spread are on average too close to 0 and 1 (slope < 1), whereas estimated probabilities with too little spread are on average too close to the prevalence (slope > 1).”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [53]
Cites
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Rao, S. J. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s263 · Cited by 1,986 · see citing works
Towards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928 · see citing works
External validation of a Cox prognostic model: principles and methods
Royston, P. J., & Altman, D. G. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-33 · Cited by 911 · see citing works
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation and Updating
Mann, C. J. H. (2009) · Kybernetes · doi:10.1108/k.2009.06738fae.002 · Cited by 794 · see citing works
Two further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556 · see citing works
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552 · see citing works
Austin, P. C., & Steyerberg, E. W. (2014) · Statistics in Medicine · doi:10.1002/sim.5941 · Cited by 362 · see citing works
Validation techniques for logistic regression models
Miller, M. E., Hui, S. L., & Tierney, W. M. (1991) · Statistics in Medicine · doi:10.1002/sim.4780100805 · Cited by 322 · see citing works
Austin, P. C., & Steyerberg, E. W. (2019) · Statistics in Medicine · doi:10.1002/sim.8281 · Cited by 311 · see citing works
McKinstry, B., Hanley, J., Wild, S., Pagliari, C., Paterson, M., Lewis, S., Sheikh, A., Krishan, A., Stoddart, A., & Padfield, P. (2013) · BMJ · doi:10.1136/bmj.f3030 · Cited by 199 · see citing works
Framework for the impact analysis and implementation of Clinical Prediction Rules (CPRs)
(Members of the International Diagnostic and Prognosis Prediction (IDAPP) group), Wallace, E., Smith, S. M., Perera, R., Vaucher, P., McCowan, C., Collins, P. G. S., Verbakel, J. Y. J., Lakhanpaul, M., & Fahey, T. P. (2011) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-11-62 · Cited by 113 · see citing works
Austin, P. C., Klaveren, D. V., Vergouwe, Y., Nieboer, D., Lee, D. S., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.05.007 · Cited by 98 · see citing works
Snell, K. I. E., Hua, H., Debray, T. P. A., Ensor, J., Look, M. P., Moons, K. G. M., & Riley, R. D. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.05.009 · Cited by 75 · see citing works
Picher, M. M., Küpcü, S., Huang, C., et al. (2013) · Lab on a Chip · doi:10.1039/c3lc41308j · Cited by 75 · see citing works
Development of Machine Learning Algorithms for Prediction of 5-Year Spinal Chordoma Survival
Karhade, A. V., Thio, Q. C. B. S., Ogink, P. T., Kim, J., Lozano‐Calderón, S. A., Raskin, K. A., & Schwab, J. H. (2018) · World Neurosurgery · doi:10.1016/j.wneu.2018.07.276 · Cited by 50 · see citing works
A dictionary of chemical engineering
(2014) · Choice Reviews Online · doi:10.5860/choice.52-0031 · Cited by 48 · see citing works
Predicting Out-of-Office Blood Pressure in the Clinic (PROOF-BP)
Sheppard, J. P., Stevens, R., Gill, P. S., Martin, U., Godwin, M. S., Hanley, J., Heneghan, C. J., Hobbs, R. F., Mant, J., McKinstry, B., et al. (2016) · Hypertension · doi:10.1161/hypertensionaha.115.07108 · Cited by 45 · see citing works
Self-citationCalibration of clinical prediction rules does not just assess bias
Vach, W. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.003 · Cited by 26 · see citing works
Prediction of life expectancy in patients with spinal epidural metastasis
Bartels, R., de Ruiter, G., Feuth, T. B., et al. (2015) · Neuro-Oncology · doi:10.1093/neuonc/nov149 · Cited by 14 · see citing works
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4
Flagged first, then by citations
Cited by
Sorted by citations
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works
Lim, C. G. Y., Chong, C. C. Y., Wong, Y. H. M., Yao, J., Ma, S., Chambers, J. C., Yeo, K. K., Tai, E. S., Tromp, J., van Dam, R. M., et al. (2026) · The Lancet Regional Health - Western Pacific · doi:10.1016/j.lanwpc.2025.101794 · Cited by 3 · see citing works
- Authors
- Stevens, R. J., & Poppe, K. K.
- Affiliations
- University of Oxford (GB)
- University of Auckland (NZ)
- Corresponding author
- Stevens, Richard J.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2020-02
- Volume
- 118
- Pages
- 93-99
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2979648592
- Cited by
- 146
- References
- 25
- Citations by year
- 2026: 31 · 2025: 23 · 2024: 23 · 2023: 14 · 2022: 29 · 2021: 23 · 2020: 3
- FWCI
- 10.75
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- National Institute for Health and Care Research
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- calibration slope, calibration, clinical prediction model, model validation, discrimination (statistics), risk prediction, statistical interpretation
- MeSH terms
- Blood Glucose, Blood Pressure Determination, Calibration, Data Interpretation, Statistical, Epidemiologic Methods, Humans, Predictive Value of Tests, Reproducibility of Results, Models, Statistical, Venous Thrombosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
