Beyond discrimination: A comparison of calibration methods and clinical usefulness of predictive models of readmission risk
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Background Prior to implementing predictive models in novel settings, analyses of calibration and clinical usefulness remain as important as discrimination, but they are not frequently discussed. Calibration is a model’s reflection of actual outcome prevalence in its predictions. Clinical usefulness refers to the utilities, costs, and harms of using a predictive model in practice. A decision analytic approach to calibrating and selecting an optimal intervention threshold may help maximize the impact of readmission risk and other preventive interventions. Objectives To select a pragmatic means of calibrating predictive models that requires a minimum amount of validation data and that performs well in practice. To evaluate the impact of miscalibration on utility and cost via clinical usefulness analyses. Materials and Methods Observational, retrospective cohort study with electronic health record data from 120,000 inpatient admissions at an urban, academic center in Manhattan. The primary outcome was thirty-day readmission for three causes: all-cause, congestive heart failure, and chronic coronary atherosclerotic disease. Predictive modeling was performed via L1-regularized logistic regression. Calibration methods were compared including Platt Scaling, Logistic Calibration, and Prevalence Adjustment. Performance of predictive modeling and calibration was assessed via discrimination (c-statistic), calibration (Spiegelhalter Z-statistic, Root Mean Square Error [RMSE] of binned predictions, Sanders and Murphy Resolutions of the Brier Score, Calibration Slope and Intercept), and clinical usefulness (utility terms represented as costs). The amount of validation data necessary to apply each calibration algorithm was also assessed. Results C-statistics by diagnosis ranged from 0.7 for all-cause readmission to 0.86 (0.78–0.93) for congestive heart failure. Logistic Calibration and Platt Scaling performed best and this difference required analyzing multiple metrics of calibration simultaneously, in particular Calibration Slopes and Intercepts. Clinical usefulness analyses provided optimal risk thresholds, which varied by reason for readmission, outcome prevalence, and calibration algorithm. Utility analyses also suggested maximum tolerable intervention costs, e.g. $1,720 for all-cause readmissions based on a published cost of readmission of $11,000. Conclusions Choice of calibration method depends on availability of validation data and on performance. Improperly calibrated models may contribute to higher costs of intervention as measured via clinical usefulness. Decision-makers must understand underlying utilities or costs inherent in the use-case at hand to assess usefulness and will obtain the optimal risk threshold to trigger intervention and intervention cost limits as a result.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kuhn, M., & Johnson, K. (2013) · doi:10.1007/978-1-4614-6849-3 · Cited by 6,788
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · cited 1×
“[1,21] Rooted in decision science and similar to the approach of “net benefit”, clinical usefulness incorporates two critical aspects of the application of any clinical predictive model to guide treatment decisions: 1) the selection of a risk threshold; 2) an understanding of the utilities, costs, and harms for the gi…”
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Bootstrap confidence intervals
DiCiccio, T. J., & Efron, B. (1996) · Statistical Science · doi:10.1214/ss/1032280214 · Cited by 2,286 · cited 1×
“RESULTS Discrimination and Calibration Performance is shown including confidence intervals (aggregated via normal intervals) for the single-cause readmissions risk models including methods of model calibration and model extension (Table 4).[49] Multiple means of calibration – logistic calibration, Platt Scaling, Preva…”
MethodTowards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
Leening, M. J. G., Vedder, M. M., Witteman, J. C. M., Pencina, M. J., & Steyerberg, E. W. (2014) · Annals of Internal Medicine · doi:10.7326/m13-1522 · Cited by 594
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552
Austin, P. C., & Steyerberg, E. W. (2014) · Statistics in Medicine · doi:10.1002/sim.5941 · Cited by 362 · cited 2×
“[1],[2] The informatics community has demonstrated significant interest in calibration of predictive models at national symposia; the need for practical assessments of approaches to calibration has been highlighted.”
BackgroundCalibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312 · cited 1×
“In other domains, like acute kidney injury prediction, for example, calibration has been shown to worsen over time.[50] Event rate shift alone was a primary reason for such drift, so practical frameworks incorporating these shifts in program evaluation are important.”
Background
Cited by
A tutorial on calibration measurements and calibration models for clinical prediction models
Huang, Y., Li, W., Macheret, F., Gabriel, R. A., & Ohno-Machado, L. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz228 · Cited by 486
Use of Machine Learning Models to Predict Death After Acute Myocardial Infarction
Khera, R., Haimovich, J., Hurley, N. C., McNamara, R., Spertus, J. A., Desai, N., Rumsfeld, J. S., Masoudi, F. A., Huang, C., Normand, S.-L., et al. (2021) · JAMA Cardiology · doi:10.1001/jamacardio.2021.0122 · Cited by 287
Guidelines on the classification of foot ulcers in people with diabetes (IWGDF 2023 update).
Monteiro‐Soares, M., Hamilton, E. J., Russell, D. A., Srisawasdi, G., Boyko, E. J., Mills, J. L., Jeffcoate, W., & Game, F. (2024) · Diabetes/Metabolism Research and Reviews · doi:10.1002/dmrr.3648 · Cited by 152
Savage, T., Wang, J., Gallo, R., Boukil, A., Patel, V., Safavi-Naini, S. A. A., Soroush, A., & Chen, J. H. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae254 · Cited by 92
Predictive Modeling for Readmission to Intensive Care: A Systematic Review
Ruppert, M. M., Loftus, T. J., Small, C., Li, H., Ozrazgat-Baslanti, T., Balch, J., Holmes, R., Tighe, P. J., Upchurch, G. R., Efron, P. A., et al. (2023) · Critical Care Explorations · doi:10.1097/cce.0000000000000848 · Cited by 24
“Additionally, calibration metrics such as Brier score, Hosmer-Lemeshow statistic, and average absolute error can further elucidate the overall performance of models in a robust manner (53, 54).”
Method
Metadata
- Authors
- Walsh, C. G., Sharman, K., & Hripcsak, G.
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2017-12
- Volume
- 76
- Pages
- 9-18
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W2766556082
- Cited by
- 99
- References
- 11
- FWCI
- 4.52
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Beyond discrimination: A comparison of calibration methods and clinical usefulness of predictive models of readmission risk. https://app.cochat.ai/papers/10.1016/j.jbi.2017.10.008
- The paper
- Walsh, C. G., Sharman, K., & Hripcsak, G. (2017). Beyond discrimination: A comparison of calibration methods and clinical usefulness of predictive models of readmission risk. Journal of Biomedical Informatics. https://doi.org/10.1016/j.jbi.2017.10.008
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.