Predictive accuracy of novel risk factors and markers: A simulation study of the sensitivity of different performance measures for the Cox proportional hazards regression model
Open accessAbstract
Predicting outcomes that occur over time is important in clinical, population health, and health services research. We compared changes in different measures of performance when a novel risk factor or marker was added to an existing Cox proportional hazards regression model. We performed Monte Carlo simulations for common measures of performance: concordance indices ( c, including various extensions to survival outcomes), Royston’s D index, R 2 -type measures, and Chambless’ adaptation of the integrated discrimination improvement to survival outcomes. We found that the increase in performance due to the inclusion of a risk factor tended to decrease as the performance of the reference model increased. Moreover, the increase in performance increased as the hazard ratio or the prevalence of a binary risk factor increased. Finally, for the concordance indices and R 2 -type measures, the absolute increase in predictive accuracy due to the inclusion of a risk factor was greater when the observed event rate was higher (low censoring). Amongst the different concordance indices, Chambless and Diao’s c-statistic exhibited the greatest increase in predictive accuracy when a novel risk factor was added to an existing model. Amongst the different R 2 -type measures, O’Quigley et al.’s modification of Nagelkerke’s R 2 index and Kent and O’Quigley’s [Formula: see text] displayed the greatest sensitivity to the addition of a novel risk factor or marker. These methods were then applied to a cohort of 8635 patients hospitalized with heart failure to examine the added benefit of a point-based scoring system for predicting mortality after initial adjustment with patient age alone.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For time-to-event outcomes, concordance is not uniquely defined and several variations of the c-statistic have been proposed that adopt different strategies to account for ties or censoring and may lead to different values for c.
“For prediction models with a time-to-event outcome, concordance is not uniquely defined and several variations of the c-statistic have been proposed.22,23 These variations typically adopt different strategies to account for ties or censoring, and may thus lead to different values for c.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270 · see citing works
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878 · see citing works
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · see citing works
A note on a general definition of the coefficient of determination
NAGELKERKE, N. J. D. (1991) · Biometrika · doi:10.1093/biomet/78.3.691 · Cited by 5,927 · see citing works
Survival analysis using sas®: a practical guide
Allison, P. D. (1995) · International Linear Collider · Cited by 3,570 · see citing works
AUC: A misleading measure of the performance of predictive distribution models
Lobo, J. M., Jiménez‐Valverde, A., & Real, R. (2008) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2007.00358.x · Cited by 3,499 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Self-citationInternal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746 · see citing works
Self-citationUno, H., Cai, T., Pencina, M. J., D'Agostino, R. B., & Wei, L. J. (2011) · Statistics in Medicine · doi:10.1002/sim.4154 · Cited by 1,535 · see citing works
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · see citing works
Self-citationPencina, M. J., & D’Agostino, R. B. (2004) · Statistics in Medicine · doi:10.1002/sim.1802 · Cited by 1,446 · see citing works
Predicting Mortality Among Patients Hospitalized for Heart Failure
Lee, D. S., Austin, P. C., Rouleau, J. L., Liu, P. P., Naimark, D. M. J., & Tu, J. V. (2003) · JAMA · doi:10.1001/jama.290.19.2581 · Cited by 1,270 · see citing works
Self-citationMeasuring classifier performance: a coherent alternative to the area under the ROC curve
Hand, D. J. (2009) · Machine Learning · doi:10.1007/s10994-009-5119-5 · Cited by 1,037 · see citing works
Generating survival times to simulate Cox proportional hazards models
Bender, R., Augustin, T., & Blettner, M. (2005) · Statistics in Medicine · doi:10.1002/sim.2059 · Cited by 892 · see citing works
Concordance probability and discriminatory power in proportional hazards regression
Gönen, M., & Heller, G. (2005) · Biometrika · doi:10.1093/biomet/92.4.965 · Cited by 685 · see citing works
Austin, P. C., & Steyerberg, E. W. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280214558972 · Cited by 516 · see citing works
Self-citationA new measure of prognostic separation in survival data
Royston, P., & Sauerbrei, W. (2004) · Statistics in Medicine · doi:10.1002/sim.1621 · Cited by 450 · cited 2× · see citing works
“if one had dichotomized the linear predictor at the sample median).(16) 2.”
BackgroundEXPLAINED VARIATION FOR LOGISTIC REGRESSION
Mittlböck, M., & SCHEMPER, M. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19961015)15:19 · Cited by 334 · see citing works
Explained Variation for Survival Models
Royston, P. (2006) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0600600105 · Cited by 319 · see citing works
Estimation of time‐dependent area under the ROC curve for long‐term risk prediction
Chambless, L. E., & Diao, G. (2005) · Statistics in Medicine · doi:10.1002/sim.2299 · Cited by 309 · see citing works
Assessment of Claims of Improved Prediction Beyond the Framingham Risk Score
Tzoulaki, I. (2009) · JAMA · doi:10.1001/jama.2009.1757 · Cited by 276 · see citing works
Austin, P. C., & Steyerberg, E. W. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-82 · Cited by 270 · see citing works
Self-citationHandbook of Statistics in Clinical Oncology
Crowley, E. B. J., & Ankerst, D. P. (2001) · Medical Entomology and Zoology · Cited by 256 · see citing works
Effectiveness of Public Report Cards for Improving the Quality of Cardiac Care
Tu, J. V., Donovan, L. R., Lee, D. S., Wang, J. T., Austin, P. C., Alter, D. A., & Ko, D. T. (2009) · JAMA · doi:10.1001/jama.2009.1731 · Cited by 256 · see citing works
Self-citationCragg, J. G., & Uhler, R. S. (1970) · Canadian Journal of Economics/Revue canadienne d économique · doi:10.2307/133656 · Cited by 244 · see citing works
SUGI supplemental library user's guide
Hastings, R. P. (1986) · SAS Institute eBooks · Cited by 231 · see citing works
Predictive Accuracy and Explained Variation in Cox Regression
SCHEMPER, M., & Henderson, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00249.x · Cited by 221 · see citing works
Measures of explained variation for survival data
Korn, E. L., & Simon, R. M. (1990) · Statistics in Medicine · doi:10.1002/sim.4780090503 · Cited by 194 · see citing works
Several methods to assess improvement in risk prediction models: Extension to survival analysis
Chambless, L. E., Cummiskey, C., & Cui, G. (2010) · Statistics in Medicine · doi:10.1002/sim.4026 · Cited by 194 · see citing works
Austin, P. C., Manca, A., Zwarenstein, M., Juurlink, D. N., & Stanbrook, M. B. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.002 · Cited by 158 · see citing works
Self-citationMeasures of dependence for censored survival data
Kent, J. T., & O’Quigley, J. (1988) · Biometrika · doi:10.1093/biomet/75.3.525 · Cited by 155 · see citing works
Explained randomness in proportional hazards models
O’Quigley, J., Xu, R., & Stare, J. (2004) · Statistics in Medicine · doi:10.1002/sim.1946 · Cited by 152 · see citing works
Janssens, A. C. J. W., Moonesinghe, R., Yang, Q., Steyerberg, E. W., van Duijn, C. M., & Khoury, M. J. (2007) · Genetics in Medicine · doi:10.1097/gim.0b013e31812eece0 · Cited by 151 · see citing works
Self-citationThe explained variation in proportional hazards regression
SCHEMPER, M. (1990) · Biometrika · doi:10.1093/biomet/77.1.216 · Cited by 127 · see citing works
Quantifying discrimination of Framingham risk functions with different survival C statistics
Pencina, M. J., D’Agostino, R. B., & Song, L. (2012) · Statistics in Medicine · doi:10.1002/sim.4508 · Cited by 95 · see citing works
Choodari‐Oskooei, B., Royston, P. J., & Parmar, M. (2011) · Statistics in Medicine · doi:10.1002/sim.4242 · Cited by 91 · see citing works
A. R.2type measure of dependence for proportional hazards models
Xu, R., & quigley, J. O. (1999) · Journal of nonparametric statistics · doi:10.1080/10485259908832799 · Cited by 61 · see citing works
Predictive capability of proportional hazards regression.
O’Quigley, J., & Flandre, P. (1994) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.91.6.2310 · Cited by 54 · see citing works
Austin, P. C., & Steyerberg, E. W. (2012) · Statistics in Medicine · doi:10.1002/sim.5598 · Cited by 47 · see citing works
Self-citationOn the prognostic value of survival models with application to gene expression signatures
Hielscher, T., Zucknick, M., Werft, W., & Benner, A. (2010) · Statistics in Medicine · doi:10.1002/sim.3768 · Cited by 38 · see citing works
Choodari‐Oskooei, B., Royston, P. J., & Parmar, M. (2012) · Statistics in Medicine · doi:10.1002/sim.5460 · Cited by 25 · see citing works
Estimation of predictive accuracy in survival analysis using R and S-PLUS
Lusa, L., Miceli, R., & Mariani, L. (2007) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2007.05.009 · Cited by 20 · see citing works
SCHEMPER, M., & Kaider, A. (1997) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(96)00045-x · Cited by 19 · see citing works
Measures of Explained Variation for a Regression Model Used in Survival Analysis
Akazawa, K. (1997) · Journal of Medical Systems · doi:10.1023/a:1022884504683 · Cited by 11 · see citing works
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4
Flagged first, then by citations
Research works printed but not linked: 1
- [30]Harrell FE The phglm procedure. SUGI supplemental library users guide, Cary, NC: SAS Institute, 1986, pp. 437– 466.
Cited by
Sorted by citations
A guide to systematic review and meta-analysis of prediction model performance
Debray, T. P. A., Damen, J. A. A. G., Snell, K. I. E., Ensor, J., Hooft, L., Reitsma, J. B., Riley, R. D., & Moons, K. G. M. (2017) · BMJ · doi:10.1136/bmj.i6460 · Cited by 657 · see citing works
A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes
Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., Collins, G. S., & Moons, K. G. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218785504 · Cited by 278 · see citing works
“For prediction models with a time-to-event outcome, concordance is not uniquely defined and several variations of the c-statistic have been proposed.(22,23) These variations typically adopt different strategies to account for ties or censoring, and may thus lead to different values for c.”
BackgroundJaspers, N. E. M., Blaha, M. J., Matsushita, K., van der Schouw, Y. T., Wareham, N. J., Khaw, K.-T., Geisel, M. H., Lehmann, N., Erbel, R., Jöckel, K.-H., et al. (2020) · European Heart Journal · doi:10.1093/eurheartj/ehz239 · Cited by 120 · see citing works
- Authors
- Austin, P. C., Pencinca, M. J., & Steyerberg, E. W.
- Affiliations
- University of Toronto (CA)
- Duke University (US)
- Erasmus MC (NL)
- Corresponding author
- Austin, Peter C
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2015-02-05
- Volume
- 26
- Issue
- 3
- Pages
- 1053-1077
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- PMC
- PMC5499735
- OpenAlex
- W2146502722
- Cited by
- 39
- References
- 46
- Citations by year
- 2026: 2 · 2024: 2 · 2023: 6 · 2022: 4 · 2021: 5 · 2020: 1 · 2019: 6 · 2018: 7 · 2017: 3 · 2016: 3
- FWCI
- 3.86
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Statistical Methods in Medical Research (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePub (Erasmus University Rotterdam) (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- prediction accuracy, risk factors, prognostic factors, Cox proportional hazards model, concordance index, integrated discrimination improvement, hazard ratio, censoring, heart failure mortality, model performance measures
- MeSH terms
- Age Factors, Heart Failure, Humans, Monte Carlo Method, Risk Factors, Biomarkers, Proportional Hazards Models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
