The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Assessing calibration of prognostic risk scores
Open accessAbstract
Current methods used to assess calibration are limited, particularly in the assessment of prognostic models. Methods for testing and visualizing calibration (e.g. the Hosmer–Lemeshow test and calibration slope) have been well thought out in the binary regression setting. However, extension of these methods to Cox models is less well known and could be improved. We describe a model-based framework for the assessment of calibration in the binary setting that provides natural extensions to the survival data setting. We show that Poisson regression models can be used to easily assess calibration in prognostic models. In addition, we show that a calibration test suggested for use in survival data has poor performance. Finally, we apply these methods to the problem of external validation of a risk score developed for the general population when assessed in a special patient population (i.e. patients with particular comorbidities, such as rheumatoid arthritis).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
General Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study
D’Agostino, R. B., Vasan, R. S., Pencina, M. J., Wolf, P. A., Cobain, M., Massaro, J. M., & Kannel, W. B. (2008) · Circulation · doi:10.1161/circulationaha.107.699579 · Cited by 7,491 · cited 4×
“Our research question is whether the Framingham risk score, which was designed to predict the risk of CVD in the general population, accurately predicts the risk of CVD in patients with RA.4,5 To examine this question, we make use of a data set of 525 patients aged 30þ years with incident RA and no prior CVD from a po…”
MethodInfluentialAssessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Statistical Methods in Cancer Research. Volume II – The Design and Analysis of Cohort Studies
Juul, S. (1989) · Scandinavian Journal of Social Medicine · doi:10.1177/140349488901700214 · Cited by 3,592
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Statistical methods in cancer research. Volume II--The design and analysis of cohort studies.
Breslow, N. E., & Day, N. E. (1987) · PubMed · Cited by 1,684
Martingale-based residuals for survival models
THERNEAU, T. M., GRAMBSCH, P. M., & FLEMING, T. R. (1990) · Biometrika · doi:10.1093/biomet/77.1.147 · Cited by 1,099
Self-citationStatistical Methods in Cancer Research, Vol. II: The Design and Analysis of Cohort Studies.
Machin, D., Breslow, N. E., & Day, N. E. (1990) · Biometrics · doi:10.2307/2532481 · Cited by 634
Multiplicative Models and Cohort Analysis
Breslow, N. E., Lubin, J. H., Marek, P., & Langholz, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477915 · Cited by 565
Two Further Applications of a Model for Binary Regression
Cox, D. R. (1958) · Biometrika · doi:10.2307/2333203 · Cited by 557
Two further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552
Chi-squared goodness-of-fit tests for the proportional hazards regression model
SCHOENFELD, D. (1980) · Biometrika · doi:10.1093/biomet/67.1.145 · Cited by 489
Evaluation of the Performance of Survival Analysis Models: Discrimination and Calibration Measures
D'Agostino, R. B., & Nam, B.-H. (2003) · Handbook of Statistics · doi:10.1016/s0169-7161(03)23001-7 · Cited by 338
The Analysis of Mortality by the Subject-Years Method
Berry, G. (1983) · Biometrics · doi:10.2307/2530817 · Cited by 278
Crowson, C. S., Matteson, E. L., Roger, V. L., Therneau, T. M., & Gabriel, S. E. (2012) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2012.03.044 · Cited by 267
Self-citationCook, N. R., Paynter, N. P., Eaton, C. B., Manson, J. E., Martin, L. W., Robinson, J. G., Rossouw, J. E., Wassertheil-Smoller, S., & Ridker, P. M. (2012) · Circulation · doi:10.1161/circulationaha.111.075929 · Cited by 263
May, S. J., & Hosmer, D. W. (1998) · Lifetime Data Analysis · doi:10.1023/a:1009612305785 · Cited by 227
A Goodness-of-Fit Test for Binary Regression Models, Based on Smoothing Methods
le Cessie, S., & van Houwelingen, J. C. (1991) · Biometrics · doi:10.2307/2532385 · Cited by 218
A method for checking regression models in survival analysis based on the risk score
Gr�nnesby, J. K., & Borgan, �. (1996) · Lifetime Data Analysis · doi:10.1007/bf00127305 · Cited by 209
A note on a goodness-of-fit test for the logistic regression model
TSIATIS, A. A. (1980) · Biometrika · doi:10.1093/biomet/67.1.250 · Cited by 169
A Global Goodness-of-Fit Statistic for the Proportional Hazards Model
Moreau, T., O’Quigley, J., & Mesbah, M. (1985) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.2307/2347465 · Cited by 138
A simple method to adjust clinical prediction models to local circumstances
Janssen, K. J. M., Vergouwe, Y., Kalkman, C. J., Grobbee, D. E., & Moons, K. G. M. (2009) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/s12630-009-9041-x · Cited by 133
Validation of probabilistic predictions
Miller, M. E., Langefeld, C. D., Tierney, W. M., Hui, S. L., & McDonald, C. J. (1993) · Medical Decision Making · doi:10.1177/0272989x9301300107 · Cited by 129
Mortality trends in rheumatoid arthritis: the role of rheumatoid factor.
Sánchez-González, Á., İçen, M., Kremers, H. M., Crowson, C. S., Davis, J. M., Therneau, T. M., Roger, V. L., & Gabriel, S. E. (2008) · PubMed · Cited by 115
Self-citationPeek, N., Arts, D. G. T., Bosman, R. J., van der Voort, P. H. J., & de Keizer, N. F. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.08.011 · Cited by 92 · cited 1×
“Limitations of the Hosmer–Lemeshow test are well known and include its dependence on arbitrary groupings of patients, poor power in small data sets as well as the fact that it only results in a p value.(1,9) A calibration plot is sometimes described as a visual representation of the Hosmer–Lemeshow test, because it ca…”
MethodA Global Goodness‐of‐Fit Statistic for Cox Regression Models
Parzen, M. I., & Lipsitz, S. R. (1999) · Biometrics · doi:10.1111/j.0006-341x.1999.00580.x · Cited by 73
Inverse Association Between Obesity and Antinuclear Antibodies in Women
González, D. A., de León, A. C., Pérez, M. D. C. R., FUENTES, R. C., Jaime, A. A., Coello, S. D., Hernández, A. G., & Díaz, B. B. (2008) · The Journal of Rheumatology · doi:10.3899/jrheum.080322 · Cited by 52
May, S. J., & Hosmer, D. W. (2004) · Lifetime Data Analysis · doi:10.1023/b:lida.0000036393.29224.1d · Cited by 47
Hosmer and Lemeshow type Goodness-of-Fit Statistics for the Cox Proportional Hazards Model
May, S. J., & Hosmer, D. W. (2003) · Handbook of statistics · doi:10.1016/s0169-7161(03)23021-2 · Cited by 37
On D. Schoenfeld's approach for testing the proportional hazards assumption
MOREAU, T., O’Quigley, J., & Lellouch, J. (1986) · Biometrika · doi:10.1093/biomet/73.2.513 · Cited by 32
Poisson models for person-years and expected rates
Atkinson, E. J., Crowson, C. S., Pedersen, R. A., & Therneau, T. M. (2008) · Cited by 18
Self-citationHarrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4<361::aid-sim168>3.0.co;2-4
Hosmer (1997) · doi:10.1002/(sici)1097-0258(19970515)16:9<965::aid-sim509>3.0.co;2-o
Cited by
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Inherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943
“A first basic goal in this literature is that the probability estimates provided by the algorithm should be well-calibrated: if the algorithm identifies a set of people as having a probability z of constituting positive instances, then approximately a z fraction of this set should indeed be positive instances [8, 14].”
BackgroundDeveloping clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544
Fairness in Recommendation Ranking through Pairwise Comparisons
Beutel, A., Chen, J., Doshi, T., Qian, H., Wei, L., Wu, Y., Heldt, L., Zhao, Z., Hong, L., Chi, E. H., et al. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330745 · Cited by 362
Metadata
- Authors
- Crowson, C. S., Atkinson, E. J., & Therneau, T. M.
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2016-07-11
- Volume
- 25
- Issue
- 4
- Pages
- 1692-1706
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W2161009717
- Cited by
- 289
- References
- 33
- FWCI
- 6.59
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.