Covariate‐adjusted measures of discrimination for survival data
Open accessAbstract
Motivation: Discrimination statistics describe the ability of a survival model to assign higher risks to individuals who experience earlier events: examples are Harrell's C‐index and Royston and Sauerbrei's D, which we call the D‐index. Prognostic covariates whose distributions are controlled by the study design (e.g. age and sex) influence discrimination and can make it difficult to compare model discrimination between studies. Although covariate adjustment is a standard procedure for quantifying disease‐risk factor associations, there are no covariate adjustment methods for discrimination statistics in censored survival data. Objective: To develop extensions of the C‐index and D‐index that describe the prognostic ability of a model adjusted for one or more covariate(s). Method: We define a covariate‐adjusted C‐index and D‐index for censored survival data, propose several estimators, and investigate their performance in simulation studies and in data from a large individual participant data meta‐analysis, the Emerging Risk Factors Collaboration. Results: The proposed methods perform well in simulations. In the Emerging Risk Factors Collaboration data, the age‐adjusted C‐index and D‐index were substantially smaller than unadjusted values. The study‐specific standard deviation of baseline age was strongly associated with the unadjusted C‐index and D‐index but not significantly associated with the age‐adjusted indices. Conclusions: The proposed estimators improve meta‐analysis comparisons, are easy to implement and give a more meaningful clinical interpretation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
It is possible to convert D values to c values by making distributional assumptions about the standard deviation of the prognostic index.
“It is also possible to convert D values to c values by making distributional assumptions with respect to the standard deviation of the prognostic index.26”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [26]
An additional advantage of IPD is that it becomes possible to adjust discrimination performance for (variation in) subject-level covariates, and thus to gain more understanding in sources of between-study heterogeneity.
“An additional advantage of IPD is that it becomes possible to adjust discrimination performance for (variation in) subject-level covariates, and thus to gain more understanding in sources of between-study heterogeneity.9,10,26,68”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965 · see citing works
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
Nishimura, R., Osako, T., Nishiyama, Y., Tashima, R., Nakano, M., Fujisue, M., Toyozumi, Y., & Arima, N. (2013) · Oncology · doi:10.1159/000353099 · Cited by 13,239 · see citing works
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · see citing works
Evaluating the Yield of Medical Tests
Harrell, F. E. (1982) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1982.03320430047030 · Cited by 3,490 · see citing works
Uno, H., Cai, T., Pencina, M. J., D'Agostino, R. B., & Wei, L. J. (2011) · Statistics in Medicine · doi:10.1002/sim.4154 · Cited by 1,535 · see citing works
Survival Model Predictive Accuracy and ROC Curves
Heagerty, P. J., & Zheng, Y. (2005) · Biometrics · doi:10.1111/j.0006-341x.2005.030814.x · Cited by 1,471 · see citing works
Pepe, M. S. (2004) · American Journal of Epidemiology · doi:10.1093/aje/kwh101 · Cited by 1,276 · see citing works
Criteria for Evaluation of Novel Markers of Cardiovascular Risk
Hlatky, M., Greenland, P., Arnett, D., Ballantyne, C., Criqui, M., Elkind, M., Go, A., Harrell, F. E. J., Hong, Y.-L., Howard, B., et al. (2009) · Circulation · doi:10.1161/circulationaha.109.192278 · Cited by 1,153 · see citing works
Concordance probability and discriminatory power in proportional hazards regression
Gönen, M., & Heller, G. (2005) · Biometrika · doi:10.1093/biomet/92.4.965 · Cited by 685 · see citing works
Prognostic Models With Competing Risks
Wolbers, M., Koller, M. T., Witteman, J. C. M., & Steyerberg, E. W. (2009) · Epidemiology · doi:10.1097/ede.0b013e3181a39056 · Cited by 628 · see citing works
Predictive value of statistical models.
Van Houwelingen, J. C., & Le Cessie, S. (1990) · Statistics in Medicine · doi:10.1002/sim.4780091109 · Cited by 621 · see citing works
Regression, Prediction and Shrinkage
Copas, J. B. (1983) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1983.tb01258.x · Cited by 612 · see citing works
A new measure of prognostic separation in survival data
Royston, P., & Sauerbrei, W. (2004) · Statistics in Medicine · doi:10.1002/sim.1621 · Cited by 450 · cited 15× · see citing works
“The adjusted D-index is an excellent alternative which is easy to compute.”
BackgroundInfluentialExploring sources of heterogeneity in systematic reviews of diagnostic tests
Lijmer, J. G., Bossuyt, P. M., & Heisterkamp, S. H. (2002) · Statistics in Medicine · doi:10.1002/sim.1185 · Cited by 446 · see citing works
Estimation of time‐dependent area under the ROC curve for long‐term risk prediction
Chambless, L. E., & Diao, G. (2005) · Statistics in Medicine · doi:10.1002/sim.2299 · Cited by 309 · see citing works
Integrating the Predictiveness of a Marker with Its Performance as a Classifier
Pepe, M. S., Feng, Z., Huang, Y., Longton, G., Prentice, R. L., Thompson, I. M., & Zheng, Y. (2007) · American Journal of Epidemiology · doi:10.1093/aje/kwm305 · Cited by 291 · see citing works
A General Regression Methodology for ROC Curve Estimation
Tosteson, A. N. A., & Begg, C. B. (1988) · Medical Decision Making · doi:10.1177/0272989x8800800309 · Cited by 274 · see citing works
Janes, H., & Pepe, M. S. (2008) · American Journal of Epidemiology · doi:10.1093/aje/kwn099 · Cited by 212 · see citing works
Hilden, J., & Gerds, T. A. (2013) · Statistics in Medicine · doi:10.1002/sim.5804 · Cited by 200 · see citing works
Reporting performance of prognostic models in cancer: a review.
Mallett, S., Royston, P., Waters, R., Dutton, S., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-21 · Cited by 195 · see citing works
The Emerging Risk Factors Collaboration (2007) · European Journal of Epidemiology · doi:10.1007/s10654-007-9165-7 · Cited by 186 · see citing works
Accommodating Covariates in Receiver Operating Characteristic Analysis
Janes, H., Longton, G., & PEPE, M. S. (2009) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0900900102 · Cited by 163 · see citing works
Model inconsistency, illustrated by the cox proportional hazards model
Ford, I., Norrie, J. D., & Ahmadi, S. (1995) · Statistics in Medicine · doi:10.1002/sim.4780140804 · Cited by 97 · see citing works
Choodari‐Oskooei, B., Royston, P. J., & Parmar, M. (2011) · Statistics in Medicine · doi:10.1002/sim.4242 · Cited by 91 · see citing works
Mihăescu, R., van Zitteren, M., van Hoek, M., Sijbrands, E. J. G., Uitterlinden, A. G., Witteman, J. C. M., Albert Hofman, D., Hunink, M. G. M., van Duijn, C. M., & Janssens, A. C. J. W. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq122 · Cited by 69 · see citing works
Assessing Risk Prediction Models Using Individual Participant Data From Multiple Studies
Pennells, L. A. M., Kaptoge, S. K., White, I. R., Thompson, S. G., Wood, A., Tipping, R., Folsom, A. R., Couper, D. J., Ballantyne, C. M., Coresh, J., et al. (2013) · American Journal of Epidemiology · doi:10.1093/aje/kwt298 · Cited by 63 · see citing works
Self-citationMeasures to assess the prognostic ability of the stratified Cox proportional hazards model
Pennells, L., White, I., Wood, A., Kaptoge, S., Sarwar, N., Kostis, J., Wilson, A., Folsom, A., Chambless, L., Benderly, M., et al. (2009) · Statistics in Medicine · doi:10.1002/sim.3378 · Cited by 53 · see citing works
Self-citationChoodari‐Oskooei, B., Royston, P. J., & Parmar, M. (2012) · Statistics in Medicine · doi:10.1002/sim.5460 · Cited by 25 · see citing works
Joint Modeling, Covariate Adjustment, and Interaction
Kerr, K. F., & PEPE, M. S. (2011) · Epidemiology · doi:10.1097/ede.0b013e31823035fb · Cited by 16 · see citing works
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4
Flagged first, then by citations
Cited by
Sorted by citations
A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes
Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., Collins, G. S., & Moons, K. G. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218785504 · Cited by 278 · see citing works
“An additional advantage of IPD is that it becomes possible to adjust discrimination performance for (variation in) subject-level covariates, and thus to gain more understanding in sources of between-study heterogeneity.(9,10,26,68) Finally, methods for meta-analyzing clinical measures of performance such as net benefi…”
- Authors
- White, I. R., Rapsomaniki, E., & for the Emerging Risk Factors Collaboration
- Affiliations
- University of Cambridge (GB)
- Farr Institute (GB)
- Corresponding author
- White, Ian R.
- Venue
- Biometrical Journal
- Publisher
- Wiley
- Published
- 2014-12-20
- Volume
- 57
- Issue
- 4
- Pages
- 592-613
- ISSN
- 0323-3847 (ISSN-L)
- 1521-4036
- PMC
- PMC4666552
- OpenAlex
- W1499379546
- Cited by
- 13
- References
- 32
- Citations by year
- 2024: 1 · 2023: 1 · 2022: 4 · 2020: 1 · 2019: 2 · 2018: 2 · 2017: 1 · 2016: 1
- FWCI
- 0.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Biometrical Journal (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- PubMed Central (acceptedVersion, cc-by)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion, cc-by)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- UCL Discovery (University College London) (submittedVersion)
- Site cant be reached (submittedVersion, cc-by)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by)
- Explore Bristol Research (submittedVersion)
- Bristol Research (University of Bristol) (submittedVersion)
- Padua Research Archive (University of Padova) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion, cc-by)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Apollo (cc-by)
- Funders
- Medical Research Council (G0700463 & U105260558)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- covariate adjustment, model discrimination, survival analysis, censored data, concordance index, D-index, prognostic model, individual patient data meta-analysis, age adjustment, risk prediction
- MeSH terms
- Analysis of Variance, Biometry, Cardiovascular Diseases, Clinical Trials as Topic, Female, Humans, Male, Middle Aged, Risk Factors, Discriminant Analysis, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
