Assessing the incremental value of diagnostic and prognostic markers: a review and illustration
Open accessAbstract
Eur J Clin Invest 2011 Background New markers may improve prediction of diagnostic and prognostic outcomes. We review various measures to quantify the incremental value of markers over standard, readily available characteristics. Methods Widely used traditional measures include the improvement in model fit or in the area under the receiver operating characteristic (ROC) curve (AUC). New measures include the net reclassification index (NRI) and decision‐analytic measures, such as the fraction of true‐positive classifications penalized for false‐positive classifications [net benefit (NB)]. For illustration, we discuss a case study on the presence of residual tumour vs. benign tissue in 544 patients with testicular cancer. We assessed three tumour markers [Alpha‐fetoprotein (AFP), Human chorionic gonadotropin (HCG) and Lactate dehydrogenase (LDH)] for their incremental value over currently standard clinical predictors. Results AUC and R2 values suggested adding continuous LDH and AFP whereas NB only favoured HCG as a potentially promising marker at a clinically defendable decision threshold of 20% risk. The NRI suggested reclassification potential of all three markers. Conclusions The improvement in standard discrimination measures, which focus on finding variables that might be promising across all decision thresholds, may not detect the most informative markers at a specific threshold of particular clinical relevance. When a marker is intended to support decision‐making, calculation of the improvement in a decision‐analytic measure, such as NB, is preferable over an overall judgment as obtained from the AUC in ROC analysis.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The reporting of studies evaluating the incremental value of specific predictors, beyond established predictors or even beyond existing prediction models, fits entirely within the remit of TRIPOD.
“The reporting of studies evaluating the incremental value of specific predictors, beyond established predictors or even beyond existing prediction models [18,42], also fits entirely within the remit of TRIPOD (see the accompanying explanation and elaboration document [43]).”
From Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement (2015) · cites this paper as [18]
An ever-increasing number of studies are evaluating the incremental value of specific predictors, beyond established predictors or even beyond existing prediction models.
“An ever-increasing number of studies are evaluating the incremental value of specific predictors, beyond established predictors or even beyond existing prediction models [18,42].”
From Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement (2015) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162 · see citing works
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581 · see citing works
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270 · see citing works
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878 · see citing works
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · see citing works
Self-citationDecision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · see citing works
Self-citationA note on a general definition of the coefficient of determination
NAGELKERKE, N. J. D. (1991) · Biometrika · doi:10.1093/biomet/78.3.691 · Cited by 5,927 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Self-citationClinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Self-citationInternal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746 · see citing works
Self-citationExtensions of net reclassification improvement calculations to measure usefulness of new biomarkers
Pencina, M. J., D’Agostino, R. B., & Steyerberg, E. W. (2010) · Statistics in Medicine · doi:10.1002/sim.4085 · Cited by 2,478 · see citing works
Self-citationDichotomizing continuous predictors in multiple regression: a bad idea
Royston, P., Altman, D. G., & Sauerbrei, W. (2006) · Statistics in Medicine · doi:10.1002/sim.2331 · Cited by 2,312 · see citing works
Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK)
McShane, L. M., Altman, D. G., Sauerbrei, W., Taube, S. E., Gion, M., & Clark, G. M. (2005) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/dji237 · Cited by 2,162 · see citing works
Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction
Cook, N. R. (2007) · Circulation · doi:10.1161/circulationaha.106.672402 · Cited by 2,123 · see citing works
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · see citing works
Self-citationThe Threshold Approach to Clinical Decision Making
Pauker, S. G., & Kassirer, J. P. (1980) · New England Journal of Medicine · doi:10.1056/nejm198005153022003 · Cited by 1,371 · see citing works
Greenland, P. (2007) · Statistics in Medicine · doi:10.1002/sim.2976 · Cited by 1,300 · see citing works
Criteria for Evaluation of Novel Markers of Cardiovascular Risk
Hlatky, M., Greenland, P., Arnett, D., Ballantyne, C., Criqui, M., Elkind, M., Go, A., Harrell, F. E. J., Hong, Y.-L., Howard, B., et al. (2009) · Circulation · doi:10.1161/circulationaha.109.192278 · Cited by 1,153 · see citing works
Regression Models in Clinical Studies: Determining Relationships Between Predictors and Response
Harre, F. E., Lee, K. L., & Pollock, B. G. (1988) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/80.15.1198 · Cited by 960 · see citing works
Prognosis and prognostic research: application and impact of prognostic models in clinical practice
Moons, K. G. M., Altman, D. G., Vergouwe, Y., & Royston, P. (2009) · BMJ · doi:10.1136/bmj.b606 · Cited by 934 · see citing works
Reilly, B. M., & Evans, A. T. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-3-200602070-00009 · Cited by 794 · see citing works
Regression Methods in Biostatistics: Linear, Logistic, Survival, and Repeated Measures Models
Lyles, R. H. (2006) · Journal of the American Statistical Association · doi:10.1198/jasa.2006.s90 · Cited by 790 · see citing works
Royston, P. J., & Sauerbrei, W. (2008) · Wiley series in probability and statistics · doi:10.1002/9780470770771 · Cited by 719 · see citing works
Regression Methods in Biostatistics
Vittinghoff, E., Glidden, D. V., Shiboski, S. C., & McCulloch, C. E. (2012) · Statistics in the health sciences · doi:10.1007/978-1-4614-1353-0 · Cited by 666 · see citing works
Multivariable Prognostic Analysis in Traumatic Brain Injury: Results from The IMPACT Study
Murray, G. D., Butcher, I., McHugh, G. S., Lü, J., Mushkudiani, N. A., Maas, A. I. R., Marmarou, A., & Steyerberg, E. W. (2007) · Journal of Neurotrauma · doi:10.1089/neu.2006.0035 · Cited by 626 · see citing works
Self-citationRoyston, P. J., & Sauerbrei, W. (2008) · Cited by 601 · see citing works
Evidence base of clinical diagnosis: Evaluation of diagnostic procedures
Knottnerus, J. A. (2002) · BMJ · doi:10.1136/bmj.324.7335.477 · Cited by 358 · see citing works
EXPLAINED VARIATION FOR LOGISTIC REGRESSION
Mittlböck, M., & SCHEMPER, M. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19961015)15:19 · Cited by 334 · see citing works
The numerical measure of the success of predictions
Peirce, C. S. (1884) · Science · doi:10.1126/science.ns-4.93.453-a · Cited by 315 · see citing works
Integrating the Predictiveness of a Marker with Its Performance as a Classifier
Pepe, M. S., Feng, Z., Huang, Y., Longton, G., Prentice, R. L., Thompson, I. M., & Zheng, Y. (2007) · American Journal of Epidemiology · doi:10.1093/aje/kwm305 · Cited by 291 · see citing works
Assessment of Claims of Improved Prediction Beyond the Framingham Risk Score
Tzoulaki, I. (2009) · JAMA · doi:10.1001/jama.2009.1757 · Cited by 276 · see citing works
Judging New Markers by Their Ability to Improve Predictive Accuracy
Kattan, M. W. (2003) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/95.9.634 · Cited by 265 · see citing works
Self-citationSteyerberg, E. W., Keizer, H. J., Fosså, S. D., et al. (1995) · Journal of Clinical Oncology · doi:10.1200/jco.1995.13.5.1177 · Cited by 254 · see citing works
Self-citationShah, T., Casas, J. P., Cooper, J. A., Tzoulaki, I., Sofat, R., McCormack, V., Smeeth, L. L., Deanfield, J. E., Lowe, G. D. O., Rumley, A., et al. (2008) · International Journal of Epidemiology · doi:10.1093/ije/dyn217 · Cited by 234 · see citing works
Predictive testing for complex diseases using multiple genes: Fact or fiction?
Janssens, A. C. J. W., Aulchenko, Y. S., Elefante, S., Borsboom, G., Steyerberg, E. W., & van Duijn, C. M. (2006) · Genetics in Medicine · doi:10.1097/01.gim.0000229689.18263.f4 · Cited by 232 · see citing works
Self-citationDecision Curve Analysis: A Discussion
Steyerberg, E. W., & Vickers, A. J. (2007) · Medical Decision Making · doi:10.1177/0272989x07312725 · Cited by 135 · see citing works
Self-citationComments on ‘Integrated discrimination and net reclassification improvements—Practical advice’
Pencina, M. J., D’Agostino, R. B., D’Agostino, R. B., & Vasan, R. S. (2007) · Statistics in Medicine · doi:10.1002/sim.3106 · Cited by 112 · see citing works
Self-citationPepe, M. S., Feng, Z., & Gu, J. W. (2007) · Statistics in Medicine · doi:10.1002/sim.2991 · Cited by 105 · see citing works
Vergouwe, Y., Steyerberg, E. W., Foster, R. S., Sleijfer, D. T., Fosså, S. D., Gerl, A., de Wit, R., Roberts, J. T., & Habbema, J. D. F. (2006) · European Urology · doi:10.1016/j.eururo.2006.06.047 · Cited by 85 · see citing works
Self-citationGreenland, S. (2007) · Statistics in Medicine · doi:10.1002/sim.2995 · Cited by 80 · see citing works
Putting Risk Prediction in Perspective: Relative Utility Curves
Baker, S. G. (2009) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djp353 · Cited by 79 · see citing works
Evaluating prediction models in reproductive medicine
Coppus, S. F. P. J., van der Veen, F., Opmeer, B. C., Mol, B. W., & Bossuyt, P. M. (2009) · Human Reproduction · doi:10.1093/humrep/dep109 · Cited by 73 · see citing works
McHugh, G. S., Butcher, I., Steyerberg, E. W., Lü, J., Mushkudiani, N. A., Marmarou, A., Maas, A. I. R., & Murray, G. D. (2007) · Journal of Neurotrauma · doi:10.1089/neu.2006.0026 · Cited by 56 · see citing works
Self-citationSteyerberg, E. W., Keizer, H. J., Stoter, G., et al. (1994) · European Journal of Cancer · doi:10.1016/0959-8049(94)90164-3 · Cited by 56 · see citing works
Self-citationExpectations, validity, and reality in omics
Ioannidis, J. P. A. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.002 · Cited by 54 · see citing works
Taylor, K. L., DiPlacido, J., Redd, W. H., Faccenda, K., Greer, L., & Perlmutter, A. P. (1999) · Cancer · doi:10.1002/(sici)1097-0142(19990315)85:6 · Cited by 50 · see citing works
Strengthening the reporting of genetic risk prediction studies (GRIPS): explanation and elaboration
Janssens, A. C. J. W., Ioannidis, J. P. A., Bedrosian, S. R., Boffetta, P., Dolan, S. M., Dowling, N. F., Fortier, I., Freedman, A. N., Grimshaw, J., Gulcher, J. R., et al. (2011) · European Journal of Clinical Investigation · doi:10.1111/j.1365-2362.2011.02493.x · Cited by 43 · see citing works
Self-citationClinical validity: Defining biomarker performance
Bossuyt, P. M. (2010) · Scandinavian Journal of Clinical and Laboratory Investigation · doi:10.3109/00365513.2010.493383 · Cited by 43 · see citing works
Henriksson, M., Palmer, S., Chen, R., Damant, J., Fitzpatrick, N. K., Abrams, K. R., Hingorani, A. D., Stenestrand, U., Janzon, M., Feder, G. S., et al. (2010) · BMJ · doi:10.1136/bmj.b5606 · Cited by 40 · see citing works
PEPE, M. S., Janes, H. E., & Gu, J. (2007) · Circulation · doi:10.1161/circulationaha.107.709253 · Cited by 30 · see citing works
Steyerberg, E. W., Marshall, P. B., Keizer, H. J., et al. (1999) · PubMed · Cited by 24 · see citing works
Self-citationGail, M. H. (2001) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2001.tb04034.x · Cited by 21 · see citing works
Applying the Lorenz curve to disease risk to optimize health benefits under cost constraints
Gail, M. H. (2009) · Statistics and Its Interface · doi:10.4310/sii.2009.v2.n2.a1 · Cited by 18 · see citing works
Strengthening the reporting of genetic risk prediction studies (GRIPS): explanation and elaboration
Janssens, A. C. J. W., Ioannidis, J. P. A., Bedrosian, S. R., Boffetta, P., Dolan, S. M., Dowling, N. F., Fortier, I., Freedman, A. N., Grimshaw, J., Gulcher, J. R., et al. (2011) · European Journal of Epidemiology · doi:10.1007/s10654-011-9551-z · Cited by 18 · see citing works
Self-citationROC curves and confidence intervals: getting them right
Altman, D. G. (2000) · Heart · doi:10.1136/heart.83.2.236 · Cited by 16 · see citing works
Chapter 1 Clinical validation of biomarkers for predicting risk
Levinson, S. S. (2009) · Advances in clinical chemistry · doi:10.1016/s0065-2423(09)48001-6 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388 · see citing works
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267 · see citing works
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. (2015) · BMC Medicine · doi:10.1186/s12916-014-0241-z · Cited by 1,927 · see citing works
Vickers, A. J., Van Calster, B., & Steyerberg, E. W. (2016) · BMJ · doi:10.1136/bmj.i6 · Cited by 1,163 · see citing works
Self-citationPrediction models for cardiovascular disease risk in the general population: systematic review
Damen, J. A. A. G., Hooft, L., Schuit, E., Debray, T. P. A., Collins, G. S., Tzoulaki, I., Lassale, C. M., Siontis, G. C. M., Chiocchia, V., Roberts, C., et al. (2016) · BMJ · doi:10.1136/bmj.i2416 · Cited by 935 · see citing works
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Circulation · doi:10.1161/circulationaha.114.014508 · Cited by 793 · see citing works
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · European Urology · doi:10.1016/j.eururo.2014.11.025 · Cited by 439 · see citing works
Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371 · see citing works
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01801-8 · Cited by 246 · see citing works
Self-citationVan Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works
Self-citationLIU, Y., MU, Y., LI, G.-Q., YU, X., LI, P.-J., SHEN, Z.-Q., WANG, H.-X., & WEI, L.-Q. (2015) · Experimental and Therapeutic Medicine · doi:10.3892/etm.2015.2678 · Cited by 8 · see citing works
- Authors
- Steyerberg, E. W., Pencina, M. J., Lingsma, H. F., Kattan, M. W., Vickers, A. J., & Van Calster, B.
- Affiliations
- Erasmus MC (NL)
- Boston University (US)
- Cleveland Clinic (US)
- Memorial Sloan Kettering Cancer Center (US)
- Venue
- European Journal of Clinical Investigation
- Publisher
- Wiley
- Published
- 2011-07-05
- Volume
- 42
- Issue
- 2
- Pages
- 216-228
- ISSN
- 0014-2972 (ISSN-L)
- 1365-2362
- PMC
- PMC3587963
- OpenAlex
- W2139621089
- Cited by
- 219
- References
- 57
- Citations by year
- 2026: 19 · 2025: 14 · 2024: 12 · 2023: 11 · 2022: 12 · 2021: 9 · 2020: 13 · 2019: 13 · 2018: 13 · 2017: 16 · 2016: 12 · 2015: 26 · 2014: 17 · 2013: 18 · 2012: 13
- FWCI
- 13.73
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- European Journal of Clinical Investigation (publishedVersion)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- Funders
- National Cancer Institute (R33CA127768, P50CA092629)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- prognostic factors, diagnostic biomarkers, net reclassification improvement, net benefit, area under the ROC curve, testicular cancer, residual tumor, alpha-fetoprotein, human chorionic gonadotropin, lactate dehydrogenase, tumor markers, clinical decision-making
- MeSH terms
- Humans, Male, Neoplasms, Germ Cell and Embryonal, Predictive Value of Tests, Prognosis, Testicular Neoplasms, Biomarkers, Tumor, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
