A new measure of prognostic separation in survival data
Abstract
Multivariable prognostic models are widely used in cancer and other disease areas, and have a range of applications in clinical medicine, clinical trials and allocation of health services resources. A well‐founded and reliable measure of the prognostic ability of a model would be valuable to help define the separation between patients or prognostic groups that the model could provide, and to act as a benchmark of model performance in a validation setting. We propose such a measure for models of survival data. Its motivation derives originally from the idea of separation between Kaplan–Meier curves. We define the criteria for a successful measure and discuss them with respect to our approach. Adjustments for ‘optimism’, the tendency for a model to predict better on the data on which it was derived than on new data, are suggested. We study the properties of the measure by simulation and by example in three substantial data sets. We believe that our new measure will prove useful as a tool to evaluate the separation available‐with a prognostic model. Copyright © 2004 John Wiley & Sons, Ltd.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Royston and Sauerbrei's D index quantifies prognostic separation of survival curves and is closely related to the standard deviation of the prognostic index.
“This index quantifies prognostic separation of survival curves and is closely related to the standard deviation of the prognostic index, with Jinks et al. suggesting an equation to convert c values to D values based on empirical evidence25”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [24]
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500 · see citing works
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,834 · see citing works
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230 · see citing works
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 21,028 · see citing works
Counting Processes and Survival Analysis.
Dabrowska, D. M., Fleming, T. R., & Harrington, D. P. (1992) · Journal of the American Statistical Association · doi:10.2307/2290673 · Cited by 2,300 · see citing works
Modeling and variable selection in epidemiologic analysis
Greenland, S. (1989) · American Journal of Public Health · doi:10.2105/ajph.79.3.340 · Cited by 2,208 · see citing works
Royston, P. J., & Parmar, M. (2002) · Statistics in Medicine · doi:10.1002/sim.1203 · Cited by 1,537 · see citing works
Self-citationStatistical Estimates and Transformed Beta-Variables.
Minton, P. D., & Blom, G. (1960) · American Mathematical Monthly · doi:10.2307/2308690 · Cited by 1,183 · see citing works
A bootstrap resampling procedure for model building: Application to the cox regression model
Sauerbrei, W., & Schumacher, M. (1992) · Statistics in Medicine · doi:10.1002/sim.4780111607 · Cited by 601 · see citing works
Self-citationAa, Evans, M., Hastings, N. A. J., & Peacock, B. (1994) · Journal of the American Statistical Association · doi:10.2307/2290968 · Cited by 426 · see citing works
Commentary: Prognostic models: clinically useful or quickly forgotten?
Wyatt, J. C., & Altman, D. G. (1995) · BMJ · doi:10.1136/bmj.311.7019.1539 · Cited by 367 · see citing works
EXPLAINED VARIATION FOR LOGISTIC REGRESSION
Mittlböck, M., & SCHEMPER, M. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19961015)15:19 · Cited by 334 · see citing works
Validation techniques for logistic regression models.
Miller, M. E., Hui, S. L., & Tierney, W. (1991) · Statistics in Medicine · doi:10.1002/sim.4780100805 · Cited by 322 · see citing works
The Use of Resampling Methods to Simplify Regression Models in Medical Statistics
Sauerbrei, W. (1999) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/1467-9876.00155 · Cited by 310 · see citing works
Self-citationPredictive Accuracy and Explained Variation in Cox Regression
SCHEMPER, M., & Henderson, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00249.x · Cited by 221 · see citing works
Inflation of R 2 in Best Subset Regression
Rencher, A. C., & Pun, F. C. (1980) · Technometrics · doi:10.1080/00401706.1980.10486100 · Cited by 162 · see citing works
Inflation of R 2 in Best Subset Regression
Rencher, A. C., & Pun, F. C. (1980) · Technometrics · doi:10.2307/1268382 · Cited by 159 · see citing works
Measures of dependence for censored survival data
Kent, J. T., & O’Quigley, J. (1988) · Biometrika · doi:10.1093/biomet/75.3.525 · Cited by 155 · see citing works
Sauerbrei, W., Hübner, K., Schmoor, C., & Schumacher, M. C. (1997) · Breast Cancer Research and Treatment · doi:10.1023/a:1005733404976 · Cited by 65 · see citing works
Self-citation- Self-citation
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4
Flagged first, then by citations
Cited by
Sorted by citations
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · see citing works
Dichotomizing continuous predictors in multiple regression: a bad idea
Royston, P., Altman, D. G., & Sauerbrei, W. (2006) · Statistics in Medicine · doi:10.1002/sim.2331 · Cited by 2,312 · see citing works
“DOI: 10.1002/sim.2331 Dichotomizing continuous predictors in multiple regression: a bad idea Patrick Royston1;∗;†, Douglas G. Altman2 and Willi Sauerbrei3 1MRC Clinical Trials Unit; 222 Euston Road; London NW1 2DA; U.K. 2Centre for Statistics in Medicine; University of Oxford; Wolfson College Annexe; Linton Road; Oxfo…”
BackgroundMethodInfluentialSelf-citationHippisley-Cox, J., Coupland, C., & Brindle, P. (2017) · BMJ · doi:10.1136/bmj.j2099 · Cited by 1,703 · see citing works
Predicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2
Hippisley-Cox, J., Coupland, C., Vinogradova, Y., Robson, J., Minhas, R., Sheikh, A., & Brindle, P. (2008) · BMJ · doi:10.1136/bmj.39609.449676.25 · Cited by 1,459 · see citing works
Riley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409 · see citing works
O'Mahony, C., Jichi, F., Pavlou, M., Monserrat, L., Anastasakis, A., Rapezzi, C., Biagini, E., Gimeno, J. R., Limongelli, G., McKenna, W. J., et al. (2014) · European Heart Journal · doi:10.1093/eurheartj/eht439 · Cited by 1,251 · see citing works
“Comparison with conventional risk factors Clinical practice guidelines consider severe hypertrophy (maximal wall thickness ≥30 mm), non-sustained ventricular tachycardia (NSVT), family history of SCD, and unexplained syncope as conventional risk factors of SCD.1,2 To compare the risk model developed in this article wi…”
MethodInfluentialHippisley-Cox, J., Coupland, C., Vinogradova, Y., Robson, J., May, M., & Brindle, P. (2007) · British medical journal · doi:10.1136/bmj.39261.471806.55 · Cited by 1,069 · see citing works
External validation of a Cox prognostic model: principles and methods
Royston, P. J., & Altman, D. G. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-33 · Cited by 911 · see citing works
“Here, for the purpose of illustration we focus on three measures: Harrell’s c-index of concordance [34], Gönen and Heller’s unbiased concordance statistic K [35], and the Royston-Sauerbrei D statistic [26].”
MethodInfluentialSelf-citationMarshall, A., Altman, D. G., HOLDER, R. L., & Royston, P. J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-57 · Cited by 883 · see citing works
Self-citationA guide to systematic review and meta-analysis of prognostic factor studies
Riley, R. D., Moons, K. G. M., Snell, K. I. E., Ensor, J., Hooft, L., Altman, D. G., Hayden, J. A., Collins, P. G. S., & Debray, T. P. A. (2019) · BMJ · doi:10.1136/bmj.k4597 · Cited by 754 · see citing works
The design of simulation studies in medical statistics
Burton, A., Altman, D. G., Royston, P. J., & HOLDER, R. L. (2006) · Statistics in Medicine · doi:10.1002/sim.2673 · Cited by 755 · see citing works
“The last considered the chosen number of simulations to be sufficient, as they were not aiming to estimate any quantities with high accuracy [ 38 ].”
BackgroundSelf-citationSwallow, E. B., Reyes, D., Hopkinson, N. S., Man, W. D.-C., Porcher, R., Cetti, E. J., Moore, A. J., Moxham, J., & Polkey, M. I. (2007) · Thorax · doi:10.1136/thx.2006.062026 · Cited by 728 · see citing works
“Additionally, the discriminative ability of different multiple models was compared using the D measure of Royston and Sauerbrei,23 as well as the D-based version of Kent and O’Quigley’s24 measure of dependence (R2 measure).”
BackgroundMethodCollins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665 · see citing works
“We examined two R(2)-type measures [40,41] (explained variation [32] and explained randomness”
BackgroundMethodInfluentialPrognostic Models With Competing Risks
Wolbers, M., Koller, M. T., Witteman, J. C. M., & Steyerberg, E. W. (2009) · Epidemiology · doi:10.1097/ede.0b013e3181a39056 · Cited by 628 · see citing works
Riley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585 · see citing works
“36 This can be interpreted as the log hazard ratio comparing two equally sized groups defined by dichotomising at the median value of the prognostic index from the developed model (where the prognostic index is defined by the combined predictor effects in the developed model, (that is, β 1 X 1 +β 2…”
BackgroundAdequate sample size for developing prediction models is not simply related to events per variable
Ogundimu, E. O., Altman, D. G., & Collins, G. S. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.02.031 · Cited by 546 · see citing works
“We examined the stability and precision of the regression coefficients and their effect on the models’ predictive performance (e.g., the c-index, D-statistic, and R2).”
InfluentialClift, A. K., Coupland, C. A. C., Keogh, R. H., Diaz‐Ordaz, K., Williamson, E., Harrison, E. M., Hayward, A. C., Hemingway, H., Horby, P., Mehta, N., et al. (2020) · BMJ · doi:10.1136/bmj.m3731 · Cited by 533 · see citing works
The effect of including C-reactive protein in cardiovascular risk prediction models for women
Cook, N. R., Buring, J. E., & Ridker, P. M. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-145-1-200607040-00128 · Cited by 504 · see citing works
“We computed the D-statistic of Royston and Sauerbrei (35), based on the separation of survival curves by predictor variables, and again adjusted for optimism.”
MethodHippisley‐Cox, J., & Coupland, C. A. C. (2009) · BMJ · doi:10.1136/bmj.b4229 · Cited by 481 · see citing works
“and compared these by 10th of predicted risk. The observed risk at 10 years was obtained by using the 10 year Kaplan-Meier estimate. We calculated the D statistic (a measure of discrimination where higher values indicate better discrimination)”
Methodsurvcomp : an R/Bioconductor package for performance assessment and comparison of survival models
Schröder, M., Culhane, A. C., Quackenbush, J. F., & Haibe‐Kains, B. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr511 · Cited by 472 · see citing works
“Although a number of model performance estimators have been described (Cox, 1972; Graf et al., 1999; Harrel et al., 1996; Heagerty et al., 2000; Royston and Sauerbrei, 2004; Verweij and Houwelingen, 1993), they are not widely used.”
Evaluation of clinical prediction models (part 2): how to undertake an external validation study
Riley, R., Archer, L., Snell, K., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L., & Collins, G. S. (2024) · British medical journal · doi:10.1136/bmj-2023-074820 · Cited by 515 · see citing works
“37 We particularly recommend reporting the Cox-Snell R 2 value, as it is needed in sample size calculations for future model development studies.”
MethodHippisley-Cox, J., Coupland, C. A. C., Robson, J., Sheikh, A., & Brindle, P. (2009) · British medical journal · doi:10.1136/bmj.b880 · Cited by 399 · see citing works
C‐reactive protein and coronary heart disease: a critical review
Casas, J. P., Shah, T., Hingorani, A. D., Danesh, J., & Pepys, M. B. (2008) · Journal of Internal Medicine · doi:10.1111/j.1365-2796.2008.02015.x · Cited by 365 · see citing works
Hippisley‐Cox, J., Coupland, C. A. C., Robson, J. P., & Brindle, P. M. (2010) · BMJ · doi:10.1136/bmj.c6624 · Cited by 365 · see citing works
Hippisley‐Cox, J., & Coupland, C. A. C. (2012) · BMJ · doi:10.1136/bmj.e3427 · Cited by 350 · see citing works
A time‐dependent discrimination index for survival data
Antolini, L., Boracchi, P., & Biganzoli, E. M. (2005) · Statistics in Medicine · doi:10.1002/sim.2427 · Cited by 340 · see citing works
“Several approaches to derive measures of predictive accuracy have been proposed: explained variation, for the extent to which the covariates determine the individual outcome [3, 5, 6]; accuracy scores, for the quantitative agreement between predicted and observed outcome [7, 8]; indexes of separation between predicted…”
BackgroundKim, L. G., Johnson, T. L., Marson, A. G., & Chadwick, D. W. (2006) · The Lancet Neurology · doi:10.1016/s1474-4422(06)70383-0 · Cited by 321 · see citing works
Explained Variation for Survival Models
Royston, P. (2006) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0600600105 · Cited by 319 · see citing works
Self-citationA framework for meta-analysis of prediction model studies with binary and time-to-event outcomes
Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., Collins, G. S., & Moons, K. G. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218785504 · Cited by 278 · see citing works
“Sometimes, discrimination is measured using Royston and Sauerbrei’s D index, usually referred to as the ‘‘D statistic’’.(24) This index quantifies prognostic separation of survival curves and is closely related to the standard deviation of the prognostic index, with Jinks et al.”
MethodPylypchuk, R. D., Wells, S., Kerr, A., Poppe, K., Riddell, T., Harwood, M. L. N., Exeter, D. J., Mehta, S., Grey, C., Wu, B., et al. (2018) · The Lancet · doi:10.1016/s0140-6736(18)30664-0 · Cited by 269 · see citing works
A gene-expression signature to predict survival in breast cancer across independent data sets
Naderi, A., Teschendorff, A. E., Barbosa‐Morais, N. L., Pinder, S. E., Green, A. R., Powe, D. G., Robertson, J. F. R., Aparicio, S. A. J. R., Ellis, I. O., Brenton, J. D., et al. (2006) · Oncogene · doi:10.1038/sj.onc.1209920 · Cited by 257 · see citing works
“Using two different prognostic classification schemes and measures, nearest centroid (HR) and risk ordering (D-index), the 70-gene classifier was also found to be prognostic in two independent external data sets.”
MethodInfluentialIncident Heart Failure Prediction in the Elderly
Butler, J., Kalogeropoulos, A. P., Georgiopoulou, V. V., BeLue, R., Rodondi, N., García, M., Bauer, D. C., Satterfield, S., Smith, A. L., Vaccarino, V., et al. (2008) · Circulation Heart Failure · doi:10.1161/circheartfailure.108.768457 · Cited by 257 · see citing works
“Glucose fasting, mg/dL 104 (34) 114 (46) 103 (33) 0.”
Kaptoge, S., Beck, T. J., Reeve, J., Stone, K. L., Hillier, T. A., Cauley, J. A., & Cummings, S. R. (2008) · Journal of Bone and Mineral Research · doi:10.1359/jbmr.080802 · Cited by 283 · see citing works
“Discriminative ability was also assessed using the D statistic,((35,36)) interpreted as the average separation of survival curves (i.”
MethodMay, M., Boulle, A., Phiri, S., Messou, E., Myer, L., Wood, R., Keiser, O., Sterne, J. A., Dabis, F., & Egger, M. (2010) · The Lancet · doi:10.1016/s0140-6736(10)60666-6 · Cited by 250 · see citing works
Collins, P. G. S., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c2442 · Cited by 250 · see citing works
Marshall, A., Altman, D. G., Royston, P. J., & HOLDER, R. L. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-7 · Cited by 248 · see citing works
“The effect of the missingness on the overall model performance was assessed using the likelihood ratio chisquare test [20], the model’s predictive ability using Nagelkerke’s R(2) statistic [20], the prognostic separation D statistic [29] and the 2-year predicted survival probability.”
Self-citationCollins, P. G. S., & Altman, D. G. (2012) · BMJ · doi:10.1136/bmj.e4181 · Cited by 245 · see citing works
Hippisley‐Cox, J., Coupland, C. A. C., Vinogradova, Y., Robson, J. P., & Brindle, P. M. (2007) · Heart · doi:10.1136/hrt.2007.134890 · Cited by 226 · see citing works
“We calculated the area under the receiver operator curve statistic to assess discrimination (ie, ability of a risk prediction equation to distinguish between those who do and do not have a cardiovascular event during the follow-up period). We also calculated the D statistic”
MethodHippisley‐Cox, J., & Coupland, C. A. C. (2017) · BMJ · doi:10.1136/bmj.j5019 · Cited by 218 · see citing works
O’Mahony, C., Jichi, F., Ommen, S. R., Christiaans, I., Arbustini, E., García‐Pavía, P., Cecchi, F., Olivotto, I., Kitaoka, H., Gotsman, I., et al. (2017) · Circulation · doi:10.1161/circulationaha.117.030437 · Cited by 216 · see citing works
“Amiodarone, n (%) 216 (7) 21 (22) 60 (10)”
Hippisley-Cox, J., & Coupland, C. (2012) · British Journal of General Practice · doi:10.3399/bjgp12x616355 · Cited by 208 · see citing works
Predictive Model for Survival in Patients With Advanced Cancer
Chow, E., Abdolell, M., Panzarella, T., Harris, K., Bezjak, A., Warde, P. R., & Tannock, I. F. (2008) · Journal of Clinical Oncology · doi:10.1200/jco.2008.17.1363 · Cited by 203 · see citing works
“These values indicate reasonable separation of the prognostic groups.(11) There were no statistically significant differences in C indexes or D statistics between training and validation sets.”
Osborn, D. P. J., Hardoon, S., Omar, R. Z., Holt, R. I. G., King, M., Larsen, J., Marston, L., Morris, R. W., Nazareth, I., Walters, K., et al. (2015) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2014.2133 · Cited by 200 · see citing works
Reporting performance of prognostic models in cancer: a review.
Mallett, S., Royston, P., Waters, R., Dutton, S., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-21 · Cited by 195 · see citing works
“Several measures of discrimination have been developed including the R squared [3], D statistic [67], c-index [4], SEP and PSEP [68,69], K [70], NRI [6], IDI [6] and decision curve analysis [71].”
MethodCollins, P. G. S., & Altman, D. G. (2009) · BMJ · doi:10.1136/bmj.b2584 · Cited by 192 · see citing works
Ioannou, G. N., Green, P. K., Kerr, K. F., & Berry, K. (2019) · Journal of Hepatology · doi:10.1016/j.jhep.2019.05.008 · Cited by 191 · see citing works
Manola, J. B., Royston, P. J., Elson, P. J., McCormack, J. B., Mazumdar, M., Négrier, S., Escudier, B. J., Eisen, T. G., Dutcher, J. P., Atkins, M. B., et al. (2011) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-11-0553 · Cited by 187 · see citing works
Self-citationCollins, G. S., Ogundimu, E. O., Cook, J. A., Manach, Y. L., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6986 · Cited by 185 · see citing works
Prediction of Symptomatic Embolism in Infective Endocarditis
Hubert, S., Thuny, F., Resseguier, N., Giorgi, R., Tribouilloy, C. M., Dolley, Y. L., Casalta, J., Ribéri, A., Chevalier, F., Ruşinaru, D., et al. (2013) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.07.029 · Cited by 183 · see citing works
“The predictive accuracy of the prognostic model was assessed by studying calibration (agreement between predicted and observed risks) and discrimination (adapted c-index (21,25) and Royston and Sauerbrei’s measure D (26) with bootstrap 95% confidence interval).”
MethodHippisley‐Cox, J., & Coupland, C. A. C. (2010) · BMC Family Practice · doi:10.1186/1471-2296-11-49 · Cited by 178 · see citing works
“We applied the algorithms obtained from the derivation cohort to both validation cohorts and calculated measures of discrimination (D statistic[26] , R2 statistic for survival data and area under the receiver operating characteristic curve (ROC statistic)) and calibration (comparing observed with predicted risks by te…”
MethodHippisley‐Cox, J., & Coupland, C. A. C. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-007825 · Cited by 177 · see citing works
“As in previous studies, 30 we calculated R 2 (explained variation in time to cancer diagnosis 31 ), D statistic 32 (a measure of discrimination where higher values indicate better discrimination) and the area under the receiver operating characteristic curve (ROC statistic) at 10 years and combined…”
MethodPrediction of Thrombo-Embolic Risk in Patients with Hypertrophic Cardiomyopathy (HCM Risk-CVA)
Guttmann, O. P., Pavlou, M., O’Mahony, C., Monserrat, L., Anastasakis, A., Rapezzi, C., Biagini, E., Gimeno, J. R., Limongelli, G., García‐Pavía, P., et al. (2015) · European Journal of Heart Failure · doi:10.1002/ejhf.316 · Cited by 165 · see citing works
Ioannou, G. N., Green, P. K., Beste, L. A., Mun, E. J., Kerr, K. F., & Berry, K. (2018) · Journal of Hepatology · doi:10.1016/j.jhep.2018.07.024 · Cited by 166 · see citing works
Ratios of involved nodes in early breast cancer
Vinh‐Hung, V., Verschraegen, C. F., Promish, D. I., Cserni, G., de Steene, J. V., Tai, P., Vlastos, G., Voordeckers, M., Storme, G. A., & Royce, M. (2004) · Breast Cancer Research · doi:10.1186/bcr934 · Cited by 162 · see citing works
“R2 represents the proportion of variation explained by covariates in regression models [35-37].”
BackgroundCardiopulmonary Exercise Testing and Prognosis in Hypertrophic Cardiomyopathy
COATS, C. J., Rantell, K. R., Bartnik, A., Patel, A., Mist, B., McKenna, W. J., & Elliot, P. M. (2015) · Circulation Heart Failure · doi:10.1161/circheartfailure.114.002248 · Cited by 161 · see citing works
THE GOLDEN AGE : MASA EFEKTIF MERANCANG KUALITAS ANAK
Uce, L. (2017) · Bunayya Jurnal Pendidikan Anak · doi:10.22373/bunayya.v1i2.1322 · Cited by 156 · see citing works
Hippisley‐Cox, J., Coupland, C. A. C., & Brindle, P. M. (2013) · BMJ · doi:10.1136/bmj.f2573 · Cited by 154 · see citing works
Utility of the Seattle Heart Failure Model in Patients With Advanced Heart Failure
Kalogeropoulos, A. P., Georgiopoulou, V. V., Giamouzis, G., Smith, A. L., Agha, S. A., Waheed, S., Laskar, S. R., Puskas, J. D., Dunbar, S. B., Vega, D., et al. (2009) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2008.10.023 · Cited by 152 · see citing works
Chang, Z., Lichtenstein, P., Larsson, H., & Fazel, S. (2015) · The Lancet Psychiatry · doi:10.1016/s2215-0366(15)00088-7 · Cited by 152 · see citing works
Marshall, A., Altman, D. G., & HOLDER, R. L. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-112 · Cited by 146 · see citing works
“each dataset was assessed in terms of the Nagelkerke’s R(2) statistic [25], the prognostic separation D statistic [26] and the 2 and 5 year predicted survival probabilities.”
MethodZivanovic, O., Leitão, M. M., Iasonos, A. E., Jacks, L. M., Zhou, Q. C., Abu‐Rustum, N. R., Soslow, R. A., Juretzka, M. M., Dennis, S., Barakat, R. R., et al. (2009) · Journal of Clinical Oncology · doi:10.1200/jco.2008.19.8366 · Cited by 140 · see citing works
Park, S. Y., Park, J. E., Kim, H., & Park, S. H. (2021) · Korean Journal of Radiology · doi:10.3348/kjr.2021.0223 · Cited by 140 · see citing works
“, and Kent and O’Quigley measure of dependence [38-42].”
BackgroundWalters, K. R., Hardoon, S. L., Petersen, I., Iliffe, S., Omar, R., Nazareth, I., & Rait, G. (2016) · BMC Medicine · doi:10.1186/s12916-016-0549-y · Cited by 138 · see citing works
“We assessed the discriminative performance of the dementia risk models by computing the Uno’s C [36] and Royston’s D [37] statistics for the validation cohort.”
MethodCescon, D. W., Haibe‐Kains, B., & Mak, T. W. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1424869112 · Cited by 132 · see citing works
“In particular, we used the D index (26) as a robust, unit-free estimate of the hazard ratio (HR).”
MethodThe current status of risk-stratified breast screening
Clift, A. K., Dodwell, D. J., Lord, S., Petrou, S., Brady, M., Collins, P. G. S., & Hippisley‐Cox, J. (2021) · British Journal of Cancer · doi:10.1038/s41416-021-01550-3 · Cited by 134 · see citing works
Hippisley‐Cox, J., & Coupland, C. A. C. (2011) · BMJ · doi:10.1136/bmj.d4656 · Cited by 130 · see citing works
“(estimated variation in time to venous thromboembolism), the D statistic”
BackgroundTillin, T., Hughes, A. D., Whincup, P., Mayet, J., Sattar, N., McKeigue, P. M., Chaturvedi, N., & On behalf of the SABRE Study Group (2014) · Heart · doi:10.1136/heartjnl-2013-304474 · Cited by 130 · see citing works
“In addition, we calculated the D statistic (a measure of separation based on the ability of the prognostic index to discriminate between participants ’ risks of an event) and R 2 statistic, 23 24 which estimates the proportion of explained variance (higher values indicate better discrimination).”
MethodRoyston, P. J., Parmar, M., & Sylvester, R. J. (2004) · Statistics in Medicine · doi:10.1002/sim.1691 · Cited by 129 · see citing works
“We refer the reader to the appendix of Reference [9] for further methodological details.”
BackgroundMethodInfluentialSelf-citationPrognostic Models: A Methodological Framework and Review of Models for Breast Cancer
Altman, D. G. (2009) · Cancer Investigation · doi:10.1080/07357900802572110 · Cited by 129 · see citing works
“Arguably, the best indicator of usefulness is the separation between risk groups in a plot showing Kaplan–Meier survival curves, which can be quantified using the D-statistic (29).”
BackgroundMethodSnell, K. I. E., Ensor, J., Debray, T. P. A., Moons, K. G. M., & Riley, R. D. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280217705678 · Cited by 130 · see citing works
“We only considered a prediction model for a binary outcome, and so further work looking at survival models is also required to examine if our recommendations generalise, and could additionally consider the D-statistic for discrimination.(43) Further work should also examine the within-study distributions after transfo…”
ResultHippisley‐Cox, J., & Coupland, C. A. C. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003482 · Cited by 123 · see citing works
Ferrat, É., Paillaud, É., Caillet, P., Laurent, M., Tournigand, C., Lagrange, J., Droz, J., Balducci, L., Audureau, É., Canouï‐Poitrine, F., et al. (2017) · Journal of Clinical Oncology · doi:10.1200/jco.2016.69.3143 · Cited by 116 · see citing works
“Discrimination (ability to separate patients with v without the outcome) was assessed using Harrell’s C-index with bootstrapped 95% CIs and the Royston-Sauerbrei D statistic (95% CI).(25) C-index values of 0.”
MethodRiley, R. D., Collins, P. G. S., Ensor, J., Archer, L., Booth, S., Mozumder, S. I., Rutherford, M. J., van Smeden, M., Lambert, P. C., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9275 · Cited by 116 · see citing works
Collins, P. G. S., Mallett, S., & Altman, D. G. (2011) · BMJ · doi:10.1136/bmj.d3651 · Cited by 114 · see citing works
“Discrimination is the ability of the risk score to differentiate between those patients who do and those who do not experience a fracture event during the study. This measure is quantified by calculating the area under the receiver operating characteristic curve statistic; a value of 0.5 represents chance and 1 repres…”
MethodInfluentialAn Evaluation of 2537 Gastrointestinal Stromal Tumors for a Proposed Clinical Staging System
Woodall, C. E. (2009) · Archives of Surgery · doi:10.1001/archsurg.2009.108 · Cited by 112 · see citing works
Cook, G., Royle, K., Pawlyn, C., Hockaday, A., Shah, V., Kaiser, M., Brown, S., Gregory, W. M., Child, J. A., Davies, F. E., et al. (2019) · The Lancet Haematology · doi:10.1016/s2352-3026(18)30220-5 · Cited by 113 · see citing works
Central Zone Carcinoma of the Prostate Gland: A Distinct Tumor Type With Poor Prognostic Features
Cohen, R. J., Shannon, B. A., Phillips, M. R., Moorin, R. E., Wheeler, T. M., & Garrett, K. L. (2008) · The Journal of Urology · doi:10.1016/j.juro.2008.01.017 · Cited by 109 · see citing works
An evaluation of penalised survival methods for developing prognostic models with rare events
Ambler, G., Seaman, S. R., & Omar, R. (2011) · Statistics in Medicine · doi:10.1002/sim.4371 · Cited by 108 · see citing works
“Calibration and discrimination were summarised visually using a plot based on prognostic separation [29].”
MethodHippisley‐Cox, J., & Coupland, C. A. C. (2011) · British Journal of General Practice · doi:10.3399/bjgp12x616346 · Cited by 109 · see citing works
Mortality trends in multiple myeloma after the introduction of novel therapies in the United States
Binder, M., Nandakumar, B. N., Rajkumar, S. V., Kapoor, P., Buadi, F. K., Dingli, D., Lacy, M. Q., Gertz, M. A., Hayman, S. R., Leung, N., et al. (2021) · Leukemia · doi:10.1038/s41375-021-01453-5 · Cited by 103 · see citing works
Hippisley‐Cox, J., & Coupland, C. A. C. (2014) · BMJ · doi:10.1136/bmj.g4606 · Cited by 95 · see citing works
Glanz, J. M., Narwaney, K. J., Mueller, S. R., Gardner, E. M., Calcaterra, S. L., Xu, S., Breslin, K. D., & Binswanger, I. A. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-017-4288-3 · Cited by 98 · see citing works
Quantifying discrimination of Framingham risk functions with different survival C statistics
Pencina, M. J., D’Agostino, R. B., & Song, L. (2012) · Statistics in Medicine · doi:10.1002/sim.4508 · Cited by 95 · see citing works
Parametric and penalized generalized survival models
Liu, X., Yudi Pawitan, Y., & Clements, M. (2016) · Statistical Methods in Medical Research · doi:10.1177/0962280216664760 · Cited by 94 · see citing works
Development and Validation of a Prognostic Nomogram for Terminally Ill Cancer Patients
Feliú, J., Jimenez-Gordo, A. M., Madero, R., Rodríguez-Aizcorbe, J. R., Espinosa, E., Castro, J., Acedo, J. D., Martínez, B., Alonso‐Babarro, A., Molina, R., et al. (2011) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djr388 · Cited by 93 · see citing works
Hippisley‐Cox, J., & Coupland, C. A. C. (2011) · British Journal of General Practice · doi:10.3399/bjgp11x606627 · Cited by 92 · see citing works
Choodari‐Oskooei, B., Royston, P. J., & Parmar, M. (2011) · Statistics in Medicine · doi:10.1002/sim.4242 · Cited by 91 · see citing works
“The graphs show that the measure proposed by Royston and Sauerbrei [7], R(2)D , is resistant to the extreme and outlier observation in the data since its mean remains approximately constant.”
BackgroundMethodResultInfluentialSelf-citationCarbohydrate antigen 125: an emerging prognostic risk factor in acute heart failure?
Núñez, J. E., Núñez, E., Consuegra, L., Sanchis, J., Bodı́, V., Martínez-Brotons, Á. M., Bertomeu‐González, V., Robles, R., Bosch, M. J., Fácila, L., et al. (2006) · Heart · doi:10.1136/hrt.2006.096016 · Cited by 90 · see citing works
Dunkler, D., Michiels, S., & SCHEMPER, M. (2007) · European Journal of Cancer · doi:10.1016/j.ejca.2006.11.018 · Cited by 89 · see citing works
Hippisley‐Cox, J., & Coupland, C. A. C. (2017) · BMJ · doi:10.1136/bmj.j4208 · Cited by 86 · see citing works
A consensus prognostic gene expression classifier for ER positive breast cancer
Teschendorff, A. E., Naderi, A., Barbosa‐Morais, N. L., Pinder, S. E., Ellis, I. O., Aparicio, S. A. J. R., Brenton, J. D., & Caldas, C. (2006) · Genome biology · doi:10.1186/gb-2006-7-10-r101 · Cited by 88 · see citing works
“The D-index measure of prognostic separa- tion, however, overcomes this difficulty since it only depends on the relative risk ordering of the external samples.”
BackgroundMethodResultInfluentialThomas, E. E., Spittal, M. J., Taxman, F. S., & Kinner, S. A. (2015) · Health & Justice · doi:10.1186/s40352-015-0022-6 · Cited by 88 · see citing works
“We computed approximate 95% confidence intervals (CIs) for Δ R 2 adj from 1000 bootstrapped samples (Royston and Sauerbrei 2004).”
BackgroundMethodRahman, M. S., Ambler, G., Choodari‐Oskooei, B., & Omar, R. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0336-2 · Cited by 89 · see citing works
“The D statistic is a discrimination measure that quantifies the observed separation between subjects with low and high predicted risk [27].”
BackgroundMethodInfluentialCollette, S., Bonnetain, F., Paolettí, X., Doffoël, M., Bouché, O., Raoul, J., Rougier, P. R., Masskouri, F., Bedenne, L., & Barbare, J. (2008) · Annals of Oncology · doi:10.1093/annonc/mdn030 · Cited by 86 · see citing works
“Harrell’s C statistics [21] and the Royston’s D statistics [22].”
BackgroundMethodThe Concordance Index C and the Mann–Whitney Parameter Pr(X>Y) with Randomly Censored Data
Koziol, J. A., & Jia, Z. (2009) · Biometrical Journal · doi:10.1002/bimj.200800228 · Cited by 85 · see citing works
“Interrelations with other statistical measures of association abound; see, for example, Schemper and Stare (1996), O’Quigley and Flandre (1994), O’Quigley, Xu, and Stare (2005), Schumacher, Graf, and Gerds (2003), Royston and Sauerbrei (2004), Newson (2006).”
BackgroundDevelopment and Validation of a Prognostic Score for Intrahepatic Cholangiocarcinoma
Raoof, M., Dumitra, S. A., Ituarte, P. H. G., Melstrom, L. G., Warner, S. G., Fong, Y., & Singh, G. P. (2017) · JAMA Surgery · doi:10.1001/jamasurg.2017.0117 · Cited by 84 · see citing works
Higher Frequency of Reoperation With a New Bicruciate-retaining Total Knee Arthroplasty
Christensen, J. C., Brothers, J. G., Stoddard, G. J., Anderson, M. B., Pelt, C. E., Gililland, J. M., & Peters, C. L. (2016) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-016-4812-5 · Cited by 83 · see citing works
“1 software (College Station, TX, USA) [22].”
InfluentialLiu, W., Laranjo, L., Klimis, H., Chiang, J., Yue, J., Marschner, S. L., Quiroz, J. C., Jorm, L. R., & Chow, C. K. (2023) · European Heart Journal - Quality of Care and Clinical Outcomes · doi:10.1093/ehjqcco/qcad017 · Cited by 84 · see citing works
Liao, W., Coupland, C. A. C., Burchardt, J., Baldwin, D., Gleeson, F. V., Baldwin, D., Batchkala, G., Buchanan, J. P., Burchardt, J., Chakraborty, R., et al. (2023) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(23)00050-4 · Cited by 79 · see citing works
- Authors
- Royston, P., & Sauerbrei, W.
- Affiliations
- MRC Clinical Trials Unit at UCL (GB)
- University Medical Center Freiburg (DE)
- Corresponding author
- Royston, Patrick
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2004-02-11
- Volume
- 23
- Issue
- 5
- Pages
- 723-748
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2159966843
- Cited by
- 450
- References
- 20
- Citations by year
- 2026: 19 · 2025: 15 · 2024: 21 · 2023: 28 · 2022: 33 · 2021: 38 · 2020: 19 · 2019: 29 · 2018: 21 · 2017: 24 · 2016: 19 · 2015: 29 · 2014: 12 · 2013: 18 · 2012: 25
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Statistics in Medicine (publishedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- FreiDok plus (Universitätsbibliothek Freiburg) (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- survival data, prognostic model, model validation, Kaplan-Meier curves, prognostic ability
- MeSH terms
- Brain Neoplasms, Breast Neoplasms, Clinical Trials as Topic, Germany, Glioma, Humans, Liver Cirrhosis, Biliary, Prognosis, Models, Statistical, Multicenter Studies as Topic, Multivariate Analysis, Survival Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
