Interpreting the concordance statistic of a logistic regression model: relation to the variance and odds ratio of a continuous explanatory variable

Austin, Peter C; Steyerberg, Ewout W

doi:10.1186/1471-2288-12-82

Cited by 271Statistical Methods in EpidemiologyMeta-analysis and systematic reviewsImbalanced Data Classification Techniques

Interpreting the concordance statistic of a logistic regression model: relation to the variance and odds ratio of a continuous explanatory variable

Open access

Austin, Peter C, Steyerberg, Ewout W·2012·BMC Medical Research Methodology

DOI 10.1186/1471-2288-12-82PMC PMC3528632

Open this paper in CoChat — free account Open PDF

Abstract

BACKGROUND: When outcomes are binary, the c-statistic (equivalent to the area under the Receiver Operating Characteristic curve) is a standard measure of the predictive accuracy of a logistic regression model. METHODS: An analytical expression was derived under the assumption that a continuous explanatory variable follows a normal distribution in those with and without the condition. We then conducted an extensive set of Monte Carlo simulations to examine whether the expressions derived under the assumption of binormality allowed for accurate prediction of the empirical c-statistic when the explanatory variable followed a normal distribution in the combined sample of those with and without the condition. We also examine the accuracy of the predicted c-statistic when the explanatory variable followed a gamma, log-normal or uniform distribution in combined sample of those with and without the condition. RESULTS: Under the assumption of binormality with equality of variances, the c-statistic follows a standard normal cumulative distribution function with dependence on the product of the standard deviation of the normal components (reflecting more heterogeneity) and the log-odds ratio (reflecting larger effects). Under the assumption of binormality with unequal variances, the c-statistic follows a standard normal cumulative distribution function with dependence on the standardized difference of the explanatory variable in those with and without the condition. In our Monte Carlo simulations, we found that these expressions allowed for reasonably accurate prediction of the empirical c-statistic when the distribution of the explanatory variable was normal, gamma, log-normal, and uniform in the entire sample of those with and without the condition. CONCLUSIONS: The discriminative ability of a continuous explanatory variable cannot be judged by its odds ratio alone, but always needs to be considered in relation to the heterogeneity of the population.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1186/1471-2288-12-82

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

ROC analysis is mathematically related to logistic regression.
“Although ROC analysis is mathematically related to logistic regression (Austin & Steyerberg, 2012)”
From Stress, stress-induced cortisol responses, and eyewitness identification performance (2016) · cites this paper as Austin & Steyerberg, 2012
A model's discrimination performance tends to deteriorate when it is applied to populations or subgroups with a more homogeneous case-mix, as there is less separation of predicted risks across individuals.
“For instance, it is well known that a model’s discrimination performance tends to deteriorate when it is applied to populations or subgroups with a more homogeneous case-mix, as there is less separation of predicted risks across individuals.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [20]
For binary outcome prediction models, the c-statistic is a reparameterization of Somer's D and can be derived from the distribution of the linear predictor, Cohen's effect size, or several correlation indices.
“For instance, for prediction models with a binary outcome, the c-statistic is a reparameterization of Somer’s D,19 and can also be derived from the distribution of the linear predictor,20 Cohen’s effect size,20 or several correlation indices.21”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [20]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
Applied logistic regression
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305 · see citing works
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270 · see citing works
Applied Logistic Regression.
Kupper, L. L., Hosmer, D. W., & Lemeshow, S. A. (1990) · Journal of the American Statistical Association · doi:10.2307/2290035 · Cited by 7,690 · see citing works
Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878 · see citing works
Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples
Austin, P. C. (2009) · Statistics in Medicine · doi:10.1002/sim.3697 · Cited by 6,758 · see citing works
Self-citation
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Self-citation
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Self-citation
The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed
Deeks, J. J., Macaskill, P., & Irwig, L. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.01.016 · Cited by 2,910 · see citing works
Statistical Methods in Diagnostic Medicine
Zhou, X., Sun, J., Pennello, G. A., Obuchowski, N. A., & McClish, D. K. (2011) · Wiley series in probability and statistics · doi:10.1002/9780470906514 · Cited by 1,785 · see citing works
Clinical Prediction Models
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · see citing works
Self-citation
The area above the ordinal dominance graph and the area below the receiver operating characteristic graph
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,458 · see citing works
Limitations of the Odds Ratio in Gauging the Performance of a Diagnostic, Prognostic, or Screening Marker
Pepe, M. S. (2004) · American Journal of Epidemiology · doi:10.1093/aje/kwh101 · Cited by 1,276 · see citing works
Validating recommendations for coronary angiography following acute myocardial infarction in the elderly: a matched analysis using propensity scores.
Normand, S., Landrum, M., Guadagnoli, E., Ayanian, J. Z., Ryan, T. J., Cleary, P., & McNeil, B. J. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00321-8 · Cited by 1,256 · see citing works
Statistical Methods in Diagnostic Medicine
Zhou, X., Obuchowski, N. A., & McClish, D. K. (2002) · Wiley series in probability and statistics · doi:10.1002/9780470317082 · Cited by 1,181 · see citing works
Standard Distance in Univariate and Multivariate Analysis
Flury, B. K., & Riedwyl, H. (1986) · The American Statistician · doi:10.1080/00031305.1986.10475403 · Cited by 289 · see citing works
Effectiveness of Public Report Cards for Improving the Quality of Cardiac Care
Tu, J. V., Donovan, L. R., Lee, D. S., Wang, J. T., Austin, P. C., Alter, D. A., & Ko, D. T. (2009) · JAMA · doi:10.1001/jama.2009.1731 · Cited by 256 · see citing works
Self-citation
External Validity of Risk Models: Use of Benchmark Values to Disentangle a Case-Mix Effect From Incorrect Coefficients
Vergouwe, Y., Moons, K. G. M., & Steyerberg, E. W. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq223 · Cited by 243 · see citing works
Self-citation
Model‐based screening by risk with application to down's syndrome
Royston, P. J., & Thompson, S. G. (1992) · Statistics in Medicine · doi:10.1002/sim.4780110211 · Cited by 176 · see citing works
The impact of genotype frequencies on the clinical validity of genomic profiling for predicting common chronic diseases
Janssens, A. C. J. W., Moonesinghe, R., Yang, Q., Steyerberg, E. W., van Duijn, C. M., & Khoury, M. J. (2007) · Genetics in Medicine · doi:10.1097/gim.0b013e31812eece0 · Cited by 151 · see citing works
Self-citation
Visualizing and assessing discrimination in the logistic regression model
Royston, P. J., & Altman, D. G. (2010) · Statistics in Medicine · doi:10.1002/sim.3994 · Cited by 111 · see citing works
Equivalence of improvement in area under ROC curve and linear discriminant analysis coefficient under assumption of normality
Demler, O., Pencina, M. J., & D’Agostino, R. B. (2011) · Statistics in Medicine · doi:10.1002/sim.4196 · Cited by 35 · see citing works

Flagged first, then by citations

Sorted by citations

Perioperative patient outcomes in the African Surgical Outcomes Study: a 7-day prospective observational cohort study
Biccard, B. M., Madiba, T. E., Kluyts, H.-L., Munlemvo, D. M., Madzimbamuto, F. D., Basenero, A., Gordon, C. S., Youssouf, C., Rakotoarison, S. R., Gobin, V., et al. (2018) · The Lancet · doi:10.1016/s0140-6736(18)30001-1 · Cited by 580 · see citing works
Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models
Austin, P. C., & Steyerberg, E. W. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280214558972 · Cited by 516 · see citing works
Self-citation
Minimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400 · see citing works
Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers
Austin, P. C., & Steyerberg, E. W. (2014) · Statistics in Medicine · doi:10.1002/sim.5941 · Cited by 362 · see citing works
Self-citation
Deep ROC Analysis and AUC as Balanced Average Accuracy, for Improved Classifier Selection, Audit and Explanation
Carrington, A., Manuel, D. G., Fieguth, P., Ramsay, T., Osmani, V., Wernly, B., Bennett, C., Hawken, S., Magwood, O., Sheikh, Y., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3145392 · Cited by 384 · see citing works
A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes
Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., Collins, G. S., & Moons, K. G. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218785504 · Cited by 278 · see citing works
“For instance, it is well known that a model’s discrimination performance tends to deteriorate when it is applied to populations or subgroups with a more homogeneous case-mix, as there is less separation of predicted risks across individuals.(9,10,20,41,42) Between-study heterogeneity may also appear when reported perf…”
Background
Intestinal congestion and right ventricular dysfunction: a link with appetite loss, inflammation, and cachexia in chronic heart failure
Valentová, M., von Haehling, S., Bauditz, J., Doehner, W., Ebner, N., Bekfani, T., Elsner, S., Sliziuk, V., Scherbakov, N., Murín, J., et al. (2016) · European Heart Journal · doi:10.1093/eurheartj/ehw008 · Cited by 232 · see citing works
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198 · see citing works
Minimum sample size for developing a multivariable prediction model using multinomial logistic regression
Pate, A., Riley, R. D., Collins, P. G. S., van Smeden, M., CALSTER, B. V., Ensor, J., & Martin, G. P. (2023) · Statistical Methods in Medical Research · doi:10.1177/09622802231151220 · Cited by 172 · see citing works
A new concordant partial AUC and partial c statistic for imbalanced data in the evaluation of machine learning algorithms
Carrington, A., Fieguth, P., Qazi, H. A., Holzinger, A., Chen, H., Mayr, F., & Manuel, D. G. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-1014-6 · Cited by 157 · see citing works
Consistency of variety of machine learning and statistical models in predicting clinical risks of individual patients: longitudinal cohort study using cardiovascular disease as exemplar
Li, Y., Sperrin, M., Ashcroft, D. M., & van Staa, T. P. (2020) · BMJ · doi:10.1136/bmj.m3919 · Cited by 141 · see citing works
Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C -statistic and calibration measures?
Snell, K. I. E., Ensor, J., Debray, T. P. A., Moons, K. G. M., & Riley, R. D. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280217705678 · Cited by 130 · see citing works
Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective
Luijken, K., Groenwold, R., van Calster, B., Steyerberg, E., & van Smeden, M. (2019) · Statistics in Medicine · doi:10.1002/sim.8183 · Cited by 103 · see citing works
Self-citation
External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb
Snell, K. I. E., Archer, L., Ensor, J., Bonnett, L. J., Debray, T. P. A., Phillips, B., Collins, P. G. S., & Riley, R. D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.011 · Cited by 125 · see citing works
Machine learning and deep learning predictive models for long-term prognosis in patients with chronic obstructive pulmonary disease: a systematic review and meta-analysis
Smith, L. A., Oakden-Rayner, L., Bird, A., Zeng, M., To, M.-S., Mukherjee, S., & Palmer, L. J. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00177-2 · Cited by 116 · see citing works
Geographic and temporal validity of prediction models: different approaches were useful to examine model performance
Austin, P. C., Klaveren, D. V., Vergouwe, Y., Nieboer, D., Lee, D. S., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.05.007 · Cited by 98 · see citing works
Self-citation
Factors influencing Internet usage in older adults (65 years and above) living in rural and urban Sweden
Berner, J., Rennemark, M., Jogréus, C., Anderberg, P., Sköldunger, A., Wahlberg, M., Elmståhl, S., & Berglund, J. S. (2014) · Health Informatics Journal · doi:10.1177/1460458214521226 · Cited by 97 · see citing works
Analogy in suffix rivalry: the case of English -ityand -ness
Arndt‐Lappe, S. (2014) · English Language and Linguistics · doi:10.1017/s136067431400015x · Cited by 93 · see citing works
Sarcopenia is Associated with Chemotherapy Toxicity in Patients Undergoing Cytoreductive Surgery with Hyperthermic Intraperitoneal Chemotherapy for Peritoneal Carcinomatosis from Colorectal Cancer
Chemama, S., Bayar, M. A., Lanoy, É., Ammari, S., Stoclin, A., Goèré, D., Elias, D. M., Raynard, B., & Antoun, S. (2016) · Annals of Surgical Oncology · doi:10.1245/s10434-016-5360-7 · Cited by 80 · see citing works
Prediction of early- and long-term mortality in adult patients acutely admitted to internal medicine: NRS-2002 and beyond
Sanson, G., Sadiraj, M., Barbin, I., Confezione, C., Matteis, D. D., Boscutti, G., Zaccari, M., & Zanetti, M. (2019) · Clinical Nutrition · doi:10.1016/j.clnu.2019.04.011 · Cited by 73 · see citing works
A note on estimating the Cox‐Snell R 2 from a reported C statistic ( AUROC ) to inform sample size calculations for developing a prediction model with a binary outcome
Riley, R. D., CALSTER, B. V., & Collins, P. G. S. (2020) · Statistics in Medicine · doi:10.1002/sim.8806 · Cited by 70 · see citing works
Acute kidney injury as an independent risk factor for unplanned 90-day hospital readmissions
Sawhney, S., Marks, A., Fluck, N., McLernon, D. J., Prescott, G. J., & Black, C. (2017) · BMC Nephrology · doi:10.1186/s12882-016-0430-4 · Cited by 67 · see citing works
Enhancing the performance of predictive models for Hospital mortality by adding nursing data
Sanson, G., Welton, J. M., Vellone, E., Cocchieri, A., Maurici, M., Zega, M., Alvaro, R., & D’Agostino, F. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.02.009 · Cited by 60 · see citing works
Estimation of required sample size for external validation of risk models for binary outcomes
Pavlou, M., Chen, Q., Omar, R., Seaman, S. R., Steyerberg, E. W., White, I. R., & Ambler, G. (2021) · Statistical Methods in Medical Research · doi:10.1177/09622802211007522 · Cited by 58 · see citing works
Self-citation
Developing more generalizable prediction models from pooled studies and large clustered data sets
de Jong, V. M. T., Moons, K. G. M., Eijkemans, M. J. C., Riley, R. D., & Debray, T. P. A. (2021) · Statistics in Medicine · doi:10.1002/sim.8981 · Cited by 56 · see citing works
Defining and Measuring a Standard Set of Patient-Relevant Outcomes in Coronary Artery Disease
Daeter, E. J., Timmermans, M. J. C., Hirsch, A., Lipsic, E., Houterman, S., van Veghel, D., van der Nat, P. B., Amoroso, G., Aydin, S., Bax, M., et al. (2018) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2018.02.037 · Cited by 53 · see citing works
Prediction of Outcome and Endovascular Treatment Benefit: Validation and Update of the MR PREDICTS Decision Tool
Venema, E., Roozenbeek, B., Mulder, M. J. H. L., Brown, S. B., Majoie, C. B. L. M., Steyerberg, E. W., Demchuk, A. M., Muir, K. W., Dávalos, A., Mitchell, P. J., et al. (2021) · Stroke · doi:10.1161/strokeaha.120.032935 · Cited by 53 · see citing works
Self-citation
Modified Brier score for evaluating prediction accuracy for binary outcomes
Yang, W., Jiang, J.-K., Schnellinger, E., Kimmel, S., & Guo, W. (2022) · Statistical Methods in Medical Research · doi:10.1177/09622802221122391 · Cited by 61 · see citing works
Utilizing Home Healthcare Electronic Health Records for Telehomecare Patients With Heart Failure
Kang, Y., McHugh, M. D., Chittams, J. L., & Bowles, K. H. (2016) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000000223 · Cited by 53 · see citing works
Informing power and sample size calculations when using inverse probability of treatment weighting using the propensity score
Austin, P. C. (2021) · Statistics in Medicine · doi:10.1002/sim.9176 · Cited by 50 · see citing works
Self-citation
Youth unemployment and economic recession in Spain: influence on health and lifestyles in young people (16–24 years old)
Aguilar‐Palacio, I., Carrera‐Lasfuentes, P., & Rabanaque, M. J. (2015) · International Journal of Public Health · doi:10.1007/s00038-015-0668-9 · Cited by 51 · see citing works
A new concordance measure for risk prediction models in external validation settings
van Klaveren, D., Gönen, M., Steyerberg, E. W., & Vergouwe, Y. (2016) · Statistics in Medicine · doi:10.1002/sim.6997 · Cited by 48 · see citing works
Self-citation
Leveraging Electronic Health Records and Machine Learning to Tailor Nursing Care for Patients at High Risk for Readmissions
Brom, H. M., Carthon, J. M. B., Ikeaba, U., & Chittams, J. L. (2019) · Journal of Nursing Care Quality · doi:10.1097/ncq.0000000000000412 · Cited by 47 · see citing works
Predictors of having a first child taken into care at birth: A population-based retrospective cohort study
Wall‐Wieler, E., Roos, L. L., Brownell, M. D., Nickel, N. C., & Chateau, D. G. (2017) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2017.09.033 · Cited by 46 · see citing works
Quantitative Identification of Maize Lodging-Causing Feature Factors Using Unmanned Aerial Vehicle Images and a Nomogram Computation
Han, L., Yang, G., Feng, H., Zhou, C., Yang, H. F., Xu, B., Li, Z., & Yang, X. (2018) · Remote Sensing · doi:10.3390/rs10101528 · Cited by 46 · see citing works
Evaluation of the ACS NSQIP Surgical Risk Calculator in Elderly Patients Undergoing Hepatectomy for Hepatocellular Carcinoma
Sahara, K., Paredes, A. Z., Merath, K., Tsilimigras, D. I., Bagante, F., Ratti, F., Marques, H. P., Soubrane, O., Beal, E. W., Lam, V. W. T., et al. (2019) · Journal of Gastrointestinal Surgery · doi:10.1007/s11605-019-04174-4 · Cited by 46 · see citing works
Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study
Damen, J. A., Debray, T. P. A., Pajouheshnia, R., Reitsma, J. B., Scholten, R. J. P. M., Moons, K. G. M., & Hooft, L. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-026160 · Cited by 45 · see citing works
Addition of Cervical Elastography May Increase Preterm Delivery Prediction Performance in Pregnant Women with Short Cervix: a Prospective Study
Park, H. S., Kwon, H., Kwak, D. W., Kim, M. Y., Seol, H., Hong, J., Shim, J., Choi, S. K., Hwang, H. S., Oh, M., et al. (2019) · Journal of Korean Medical Science · doi:10.3346/jkms.2019.34.e68 · Cited by 43 · see citing works
Surface water detection in the Caucasus
Worden, J., & de Beurs, K. M. (2020) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2020.102159 · Cited by 43 · see citing works
Academic Proficiency in Children After Early Congenital Heart Disease Surgery
Mulkey, S. B., Swearingen, C. J., Melguizo, M. S., Reeves, R. N., Rowell, J. A., Gibson, N., Holland, G., Bhutta, A. T., & Kaiser, J. R. (2013) · Pediatric Cardiology · doi:10.1007/s00246-013-0781-6 · Cited by 42 · see citing works
Few promising multivariable prognostic models exist for recovery of people with non-specific neck pain in musculoskeletal primary care: a systematic review
Wingbermühle, R. W., van Trijffel, E., Nelissen, P. M., Koes, B. W., & Verhagen, A. P. (2017) · Journal of physiotherapy · doi:10.1016/j.jphys.2017.11.013 · Cited by 42 · see citing works
Evaluating preterm care across Europe using the eNewborn European Network database
Haumont, D., Modi, N., Saugstad, O. D., Antetere, R., NguyenBa, C., Turner, M. A., Costeloe, K., & Aelvoet, W. (2020) · Pediatric Research · doi:10.1038/s41390-020-0769-x · Cited by 40 · see citing works
How variation in predictor measurement affects the discriminative ability and transportability of a prediction model
Pajouheshnia, R., van Smeden, M., Peelen, L. M., & Groenwold, R. H. H. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.09.001 · Cited by 40 · see citing works
The Predictive Performance of a Pneumonia Severity Score in Human Immunodeficiency Virus–negative Children Presenting to Hospital in 7 Low- and Middle-income Countries
Gallagher, K. E., Knoll, M. D., Prosperi, C., Baggett, H. C., Brooks, W. A., Feikin, D. R., Hammitt, L. L., Howie, S. R. C., Kotloff, K. L., Levine, O. S., et al. (2019) · Clinical Infectious Diseases · doi:10.1093/cid/ciz350 · Cited by 40 · see citing works
Assessing Discriminative Performance at External Validation of Clinical Prediction Models
Nieboer, D., van der Ploeg, T., & Steyerberg, E. W. (2016) · PLoS ONE · doi:10.1371/journal.pone.0148820 · Cited by 40 · see citing works
Self-citation
Predictive accuracy of novel risk factors and markers: A simulation study of the sensitivity of different performance measures for the Cox proportional hazards regression model
Austin, P. C., Pencinca, M. J., & Steyerberg, E. W. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280214567141 · Cited by 39 · see citing works
Self-citation
Perceived discrimination and self-rated health in Canada: an exploratory study
Mont, J. D., & Forte, T. (2016) · BMC Public Health · doi:10.1186/s12889-016-3344-y · Cited by 39 · see citing works
Intradialytic Complement Activation Precedes the Development of Cardiovascular Events in Hemodialysis Patients
Poppelaars, F., da Costa, M. G., Faria, B., Berger, S. P., Assa, S., Daha, M. R., Pestana, J. O. M., van Son, W. J., Franssen, C. F. M., & Seelen, M. A. (2018) · Frontiers in Immunology · doi:10.3389/fimmu.2018.02070 · Cited by 39 · see citing works
Stress, stress-induced cortisol responses, and eyewitness identification performance
Sauerland, M., Raymaekers, L. H. C., Otgaar, H., Memon, A., Waltjen, T. T., Nivo, M., Slegers, C., Broers, N. J., & Smeets, T. (2016) · Behavioral Sciences & the Law · doi:10.1002/bsl.2249 · Cited by 37 · see citing works
Strong predictive value of mannose-binding lectin levels for cardiovascular risk of hemodialysis patients
Poppelaars, F., da Costa, M. G., Berger, S. P., Assa, S., Meter-Arkema, A. H., Daha, M. R., van Son, W. J., Franssen, C. F. M., & Seelen, M. A. (2016) · Journal of Translational Medicine · doi:10.1186/s12967-016-0995-5 · Cited by 38 · see citing works
Accounting for treatment use when validating a prognostic model: a simulation study
Pajouheshnia, R., Peelen, L. M., Moons, K. G. M., Reitsma, J. B., & Groenwold, R. H. H. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0375-8 · Cited by 38 · see citing works
Artificial intelligence in the detection, characterisation and prediction of hepatocellular carcinoma: a narrative review
Kawka, M., Dawidziuk, A., Jiao, L. R., & Gall, T. (2021) · Translational Gastroenterology and Hepatology · doi:10.21037/tgh-20-242 · Cited by 38 · see citing works
Revisit of risk factors for major obstetric hemorrhage: insights from a large medical center
Helman, S., Drukker, L., Fruchtman-Brot, H., Ioscovich, A., Farkash, R., Avitan, T., Samueloff, A., & Grisaru‐Granovsky, S. (2015) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-015-3725-y · Cited by 36 · see citing works
Maternal detection of neonatal jaundice during birth hospitalization using a novel two-color icterometer
Olusanya, B. O., Slusher, T. M., Imosemi, D. O., & Emokpae, A. A. (2017) · PLoS ONE · doi:10.1371/journal.pone.0183882 · Cited by 36 · see citing works
Combined Biomarkers Predict Acute Mortality Among Critically Ill Patients with Suspected Sepsis
Kelly, B. J., Lautenbach, E., Nachamkin, I., Coffin, S., Gerber, J., Fuchs, B., Garrigan, C., Han, X.-Y., Bilker, W., Wise, J., et al. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000003137 · Cited by 37 · see citing works
Logistic regression and machine learning predicted patient mortality from large sets of diagnosis codes comparably
Cowling, T. E., Cromwell, D. A., Bellot, A., Sharples, L. D., & van der Meulen, J. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.12.018 · Cited by 36 · see citing works
Psychosocial Determinants of Hand Hygiene, Facemask Wearing, and Physical Distancing During the COVID-19 Pandemic: A Systematic Review and Meta-analysis
Liang, W., Duan, Y., Li, F., Rhodes, R. E., Wang, X., Peiris, D. L. I. H. K., Zhou, L., Shang, B., Yang, Y., Baker, J. S., et al. (2022) · Annals of Behavioral Medicine · doi:10.1093/abm/kaac049 · Cited by 36 · see citing works
Predictive modeling of the outcomes of chemotherapy-induced (febrile) neutropenia prophylaxis with biosimilar filgrastim (MONITOR-GCSF study)
Aapro, M., Ludwig, H., Bokemeyer, C., Gascón, P., Boccadoro, M., Denhaerynck, K., Krendyukov, A., Gorray, M., MacDonald, K., & Abraham, I. L. (2016) · Annals of Oncology · doi:10.1093/annonc/mdw309 · Cited by 35 · see citing works
Longitudinal Fairness with Censorship
Zhang, W., & Weiss, J. C. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21484 · Cited by 37 · see citing works
Self-rated health: analysis of distances and transitions between response options
Perneger, T., Gayet‐Ageron, A., Courvoisier, D. S., Agoritsas, T., & Cullati, S. (2013) · Quality of Life Research · doi:10.1007/s11136-013-0418-5 · Cited by 33 · see citing works
Validating performance of TRISS, TARN and NORMIT survival prediction models in a Norwegian trauma population
Skaga, N. O., Eken, T., & Søvik, S. (2017) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.13029 · Cited by 31 · see citing works
Self-reported myocardial infarction and fatal coronary heart disease among oil spill workers and community members 5 years after Deepwater Horizon
Strelitz, J., Keil, A. P., Richardson, D. B., HEISS, G. M., Gammon, M. D., Kwok, R. K., Sandler, D. P., & Engel, L. S. (2018) · Environmental Research · doi:10.1016/j.envres.2018.09.026 · Cited by 31 · see citing works
Current Risk Adjustment and Comorbidity Index Underperformance in Predicting Post‐Acute Utilization and Hospital Readmissions After Joint Replacements: Implications for Comprehensive Care for Joint Replacement Model
Kumar, A., Karmarkar, A. M., Downer, B. G., Vashist, A., Adhikari, D., Snih, S. A., & Ottenbacher, K. J. (2017) · Arthritis Care & Research · doi:10.1002/acr.23195 · Cited by 30 · see citing works
Predictors of Patient Satisfaction After Refractive Lens Exchange With an Extended Depth of Focus IOL
Schallhorn, S. C., Hettinger, K. A., Teenan, D., Venter, J. A., Hannan, S. J., & Schallhorn, J. M. (2020) · Journal of Refractive Surgery · doi:10.3928/1081597x-20200211-01 · Cited by 30 · see citing works
Prognostic Tools for Early Mortality in Hemorrhagic Stroke: Systematic Review and Meta-Analysis
Mattishent, K., Kwok, C. S., Ashkir, L., Pelpola, K., Myint, P. K., & Loke, Y. K. (2015) · Journal of Clinical Neurology · doi:10.3988/jcn.2015.11.4.339 · Cited by 30 · see citing works
Close encounters of the fatal kind: Landscape features associated with central mountain caribou mortalities
McKay, T. L., Pigeon, K. E., Larsen, T. A., & Finnegan, L. A. (2021) · Ecology and Evolution · doi:10.1002/ece3.7190 · Cited by 28 · see citing works
Fibroblast Growth Factor 23 and Kidney Disease Progression in Autosomal Dominant Polycystic Kidney Disease
Chonchol, M., Gitomer, B. Y., Isakova, T., Cai, X., Salusky, I. B., Pereira, R. C., Abebe, K. Z., Torres, V. E., Steinman, T. I., Grantham, J. J., et al. (2017) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.12821216 · Cited by 28 · see citing works
Development and Internal Validation of a Multivariable Prediction Model for Individual Episodic Migraine Attacks Based on Daily Trigger Exposures
Holsteen, K., Hittle, M., Barad, M. J., & Nelson, L. M. (2020) · Headache The Journal of Head and Face Pain · doi:10.1111/head.13960 · Cited by 27 · see citing works
Having a usual source of care and its associated factors in Korean adults: a cross-sectional study of the 2012 Korea Health Panel Survey
An, A. R., Kim, K., Lee, J., Sung, N.-J., Lee, S., & Hyun, M. K. (2016) · BMC Family Practice · doi:10.1186/s12875-016-0555-3 · Cited by 26 · see citing works
Psychosocial Variables Contribute to Length of Stay and Discharge Destination After Lumbar Surgery Independent of Demographic and Clinical Variables
Mancuso, C. A., Duculan, R., Craig, C. M., & Girardi, F. P. (2017) · Spine · doi:10.1097/brs.0000000000002312 · Cited by 25 · see citing works
Weekends affect mortality risk and chance of discharge in critically ill patients: a retrospective study in the Austrian registry for intensive care
Zajic, P., Bauer, P., Rhodes, A., Moreno, R. P., Fellinger, T., Metnitz, B., Stavropoulou, F., Posch, M., & Metnitz, P. (2017) · Critical Care · doi:10.1186/s13054-017-1812-0 · Cited by 25 · see citing works
E-mail communication patterns and job burnout
Estévez-Mujica, C. P., & Quintane, E. (2018) · PLoS ONE · doi:10.1371/journal.pone.0193966 · Cited by 28 · see citing works
Combined biomarkers discriminate a low likelihood of bacterial infection among surgical intensive care unit patients with suspected sepsis
Kelly, B. J., Lautenbach, E., Nachamkin, I., Coffin, S. E., Gerber, J. S., Fuchs, B. D., Garrigan, C., Han, X., Bilker, W. B., Wise, J. A., et al. (2016) · Diagnostic Microbiology and Infectious Disease · doi:10.1016/j.diagmicrobio.2016.01.003 · Cited by 24 · see citing works
Knowledge and Perceptions of Tobacco-Related Harm Associated with Intention to Quit among Cigarette Smokers, e-Cigarette Users, and Dual Users: Findings from the US Population Assessment of Tobacco and Health (PATH) Wave 1
Lin, W., Martinez, S., Ding, K., & Beebe, L. (2021) · Substance Use & Misuse · doi:10.1080/10826084.2021.1879145 · Cited by 23 · see citing works
Does a Large Infant Head or a Short Perineal Body Increase the Risk of Obstetrical Perineal Trauma?
Komorowski, L. K., Leeman, L. M., Fullilove, A. M., Bedrick, E. J., Migliaccio, L., & Rogers, R. G. (2014) · Birth · doi:10.1111/birt.12101 · Cited by 24 · see citing works
Fundamentals of Clinical Prediction Modeling for the Neurosurgeon
Mijderwijk, H.-J., Steyerberg, E. W., Steiger, H., Fischer, I., & Kamp, M. A. (2019) · Neurosurgery · doi:10.1093/neuros/nyz282 · Cited by 23 · see citing works
Self-citation
Effect of Provider Volume on the Accuracy of Hospital Report Cards
Austin, P. C., & Reeves, M. J. (2014) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.113.000685 · Cited by 22 · see citing works
Self-citation
Predictors of Satisfactory Outcomes With Fixed-Bearing Lateral Unicompartmental Knee Arthroplasty: Up to 7-year Follow-Up
Xue, H., Ma, T., Wen, T., Yang, T., Xue, L., & Tu, Y. (2020) · The Journal of Arthroplasty · doi:10.1016/j.arth.2020.10.001 · Cited by 22 · see citing works
A Model to Predict Treatment Failure in Patients Undergoing Upfront Surgery for Resectable Colorectal Liver Metastases
Berardi, G., Chou, J. F., Gönen, M., Balachandran, V. P., Drebin, J. A., Jarnagin, W. R., Kingham, T. P., Soares, K. C., Wei, A. C.-C., & D'Angelica, M. I. (2023) · Annals of Surgical Oncology · doi:10.1245/s10434-023-13113-2 · Cited by 21 · see citing works
Quantifying performance of a diagnostic test as the expected information for discrimination: Relation to the C -statistic
McKeigue, P. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280218776989 · Cited by 21 · see citing works
Determining the Independent Risk Factors for Worse SCAR-Q Scores and Future Scar Revision Surgery
Ziolkowski, N. I., Behman, R. S., Klassen, A. F., Fish, J. S., Mundy, L. R., She, R. W., Forrest, C. R., Hollenbeck, S. T., Arriagada, C., Greenhalgh, D., et al. (2021) · Plastic & Reconstructive Surgery · doi:10.1097/prs.0000000000008095 · Cited by 21 · see citing works
Development and internal validation of a nomogram for predicting outcomes in children with traumatic subdural hematoma
Kaewborisutsakul, A., & Tunthanathip, T. (2022) · Acute and Critical Care · doi:10.4266/acc.2021.01795 · Cited by 21 · see citing works
Development and validation of a Medicines Comorbidity Index for older people
Narayan, S. W., & Nishtala, P. S. (2017) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-017-2333-0 · Cited by 21 · see citing works
The Effect of Increasing Age on Outcomes of Digital Revascularization or Replantation
Retrouvey, H., Solaja, O., & Baltzer, H. L. (2018) · Plastic & Reconstructive Surgery · doi:10.1097/prs.0000000000005195 · Cited by 21 · see citing works
Prognostic risk models for incident hypertension: A PRISMA systematic review and meta-analysis
Schjerven, F. E., Lindseth, F., & Steinsland, I. (2024) · PLoS ONE · doi:10.1371/journal.pone.0294148 · Cited by 23 · see citing works
Downgrading of grade group 2 intermediate‐risk prostate cancer from biopsy to radical prostatectomy: Comparison of outcomes and predictors to identify potential candidates for active surveillance
Su, Z. T., Patel, H. D., Epstein, J. I., Pavlovich, C. P., & Allaf, M. E. (2020) · Cancer · doi:10.1002/cncr.32709 · Cited by 20 · see citing works
Timing of Adverse Prostate Cancer Reclassification on First Surveillance Biopsy: Results from the Canary Prostate Cancer Active Surveillance Study
Macleod, L. C., Ellis, W. J., Newcomb, L. F., Zheng, Y., Brooks, J. D., Carroll, P. R., Gleave, M. E., Lance, R. S., Nelson, P. S., Thompson, I. M., et al. (2016) · The Journal of Urology · doi:10.1016/j.juro.2016.10.090 · Cited by 20 · see citing works
Impact of correlation of predictors on discrimination of risk models in development and external populations
Kundu, S., Mazumdar, M., & Ferket, B. S. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0345-1 · Cited by 21 · see citing works
Critical loads of atmospheric N deposition for phytoplankton nutrient limitation shifts in western U.S. mountain lakes
Williams, J. J., Lynch, J., Saros, J. E., & Labou, S. G. (2017) · Ecosphere · doi:10.1002/ecs2.1955 · Cited by 20 · see citing works
Heterogeneity in coronary heart disease risk
Simonetto, C., Rospleszcz, S., Kaiser, J. C., & Furukawa, K. (2022) · Scientific Reports · doi:10.1038/s41598-022-14013-3 · Cited by 19 · see citing works
Failure-to-Rescue After Acute Myocardial Infarction
Silber, J. H., Arriaga, A. F., Niknam, B. A., Hill, A. S., Ross, R. N., & Romano, P. S. (2018) · Medical Care · doi:10.1097/mlr.0000000000000904 · Cited by 19 · see citing works
Prediction model for short-term mortality after palliative radiotherapy for patients having advanced cancer: a cohort study from routine electronic medical data
Lee, S. F., Luk, H., Wong, A., Ng, C. K., Wong, F. C. S., & Luque-Fernández, M. Á. (2020) · Scientific Reports · doi:10.1038/s41598-020-62826-x · Cited by 19 · see citing works
Interpretable machine learning models to predict short-term postoperative outcomes following posterior cervical fusion
Karabacak, M., & Margetis, K. D. (2023) · PLoS ONE · doi:10.1371/journal.pone.0288939 · Cited by 18 · see citing works
Improving Current Glycated Hemoglobin Prediction in Adults: Use of Machine Learning Algorithms With Electronic Health Records
Alhassan, Z., Watson, M., Budgen, D., Alshammari, R., Alessa, A., & Moubayed, N. A. (2021) · JMIR Medical Informatics · doi:10.2196/25237 · Cited by 18 · see citing works
Assisting schizophrenia diagnosis using clinical electroencephalography and interpretable graph neural networks: a real-world and cross-site study
Jiang, H., Chen, P., Sun, Z., Liang, C., Xue, R., Zhao, L., Wang, Q., Li, X., Deng, W., Gao, Z., et al. (2023) · Neuropsychopharmacology · doi:10.1038/s41386-023-01658-5 · Cited by 18 · see citing works
Predictive Accuracy of 29-Comorbidity Index for In-Hospital Deaths in US Adult Hospitalizations with a Diagnosis of Venous Thromboembolism
Tsai, J. H., Abe, K., Boulet, S. L., Beckman, M. G., Hooper, W. C., & Grant, A. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0070061 · Cited by 18 · see citing works
A longitudinal cohort study of malaria exposure and changing serostatus in a malaria endemic area of rural Tanzania
Simmons, R., Mboera, L. E. G., Miranda, M. L., Morris, A. M., Stresman, G. H., Turner, E. L., Kramer, R. A., DRAKELEY, C. J., & O’Meara, W. P. (2017) · Malaria Journal · doi:10.1186/s12936-017-1945-2 · Cited by 18 · see citing works
Early Survival and Growth Plasticity of 33 Species Planted in 38 Arboreta across the European Atlantic Area
Correia, A., Almeida, M. H., Branco, M., Tomé, M., Montoya, R. C., Lucchio, L. D., Cantero, A., Díez, J. J., Prieto-Recio, C., Bravo, F., et al. (2018) · Forests · doi:10.3390/f9100630 · Cited by 18 · see citing works
Predicting kidney failure risk after acute kidney injury among people receiving nephrology clinic care
Sawhney, S., Beaulieu, M., Black, C., Djurdjev, O., Espino-Hernández, G., Marks, A., McLernon, D. J., Sheriff, Z., & Levin, A. (2018) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfy294 · Cited by 17 · see citing works
On Predicting lung cancer subtypes using ‘omic’ data from tumor and tumor-adjacent histologically-normal tissue
Pineda, A. L., Ogoe, H., Balasubramanian, J. B., Rangel‐Escareño, C., Visweswaran, S., Herman, J. G., & Gopalakrishnan, V. (2016) · BMC Cancer · doi:10.1186/s12885-016-2223-3 · Cited by 17 · see citing works

Review and evaluation of the concordance measures for assessing discrimination in the logistic regression methods
Biswas, B. N., Husain, M., & Rahman, M. S. (2019) · Journal of Statistical Research · doi:10.47302/jsr.2019530104 · Cited by 3 · see citing works
Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C -statistic and calibration measures?
Snell, K. I. E., Ensor, J., Debray, T. P. A., Moons, K. G. M., & Riley, R. D. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280217705678 · Cited by 130 · see citing works
Properties ofR2 statistics for logistic regression
Hu, B., Palta, M., & Shao, J. (2005) · Statistics in Medicine · doi:10.1002/sim.2300 · Cited by 61 · see citing works
Logistic modelling of normality by Receiver operating characteristic curves
Lee, J.-Y., & Rhee, S.-W. (2003) · Journal of Interdisciplinary Mathematics · doi:10.1080/09720502.2003.10700336
On comparisons of normal and logistic models in the bivariate dichotomous analysis
Lee, L. (1979) · Economics Letters · doi:10.1016/0165-1765(79)90226-x · Cited by 4 · see citing works

Authors: Austin, P. C., & Steyerberg, E. W.
Affiliations: Institute for Clinical Evaluative Sciences (CA)
Erasmus MC (NL)
Corresponding author: Austin, Peter C
ORCID: Austin, Peter C
Steyerberg, Ewout W
Venue: BMC Medical Research Methodology
Publisher: Springer Science and Business Media LLC
Published: 2012-06-20
Volume: 12
Issue: 1
Pages: 82-82
ISSN: 1471-2288 (ISSN-L)
DOI: 10.1186/1471-2288-12-82
PMC: PMC3528632
OpenAlex: W2112200626
Cited by: 270
References: 23
Citations by year: 2026: 17 · 2025: 16 · 2024: 25 · 2023: 29 · 2022: 26 · 2021: 17 · 2020: 30 · 2019: 20 · 2018: 19 · 2017: 28 · 2016: 17 · 2015: 12 · 2014: 7 · 2013: 7
FWCI: 7.93
Type: journal article
Language: en
Open access: gold
doi.org
Licence: cc-by
creativecommons.org/licenses/by/2.0 (tdm)
Locations: BMC Medical Research Methodology (publishedVersion, cc-by)
Europe PMC (PubMed Central) (publishedVersion, cc-by)
PubMed Central (publishedVersion, cc-by)
PubMed (publishedVersion)
EUR Research Repository (Erasmus University Rotterdam) (acceptedVersion, other-oa)
TSpace (University of Toronto) (submittedVersion)
EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders: Heart and Stroke Foundation of Canada
Institute for Clinical Evaluative Sciences
Ontario Ministry of Health and Long-Term Care
Canadian Institutes of Health Research (86508, MOP86508)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
Keywords: area under the ROC curve, logistic regression, odds ratio, heteroscedasticity, prediction accuracy, discriminatory power, non-normal distributions, gamma distribution, lognormal distribution, uniform distribution
MeSH terms: Age Factors, Analysis of Variance, Data Interpretation, Statistical, Humans, Monte Carlo Method, Myocardial Infarction, ROC Curve, Logistic Models, Odds Ratio

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Interpreting the concordance statistic of a logistic regression model: relation to the variance and odds ratio of a continuous explanatory variable

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Self-check

Cites

Cited by

Related

Record

About these checks