Predictive value of statistical models.

Van Houwelingen, J. C.; Le Cessie, S.

doi:10.1002/sim.4780091109

Cited by 621Statistical Methods in EpidemiologyStatistical Methods and InferenceStatistical Methods and Applications

Predictive value of statistical models.

Van Houwelingen, J. C., Le Cessie, S.·1990·Statistics in Medicine

DOI 10.1002/sim.4780091109

Open this paper in CoChat — free account

Abstract

A review is given of different ways of estimating the error rate of a prediction rule based on a statistical model. A distinction is drawn between apparent, optimum and actual error rates. Moreover it is shown how cross-validation can be used to obtain an adjusted predictor with smaller error rate. A detailed discussion is given for ordinary least squares, logistic regression and Cox regression in survival analysis. Finally, the splitsample approach is discussed and demonstrated on two data sets.

Integrity signals

Registry

Crossref · 10.1002/sim.4780091109

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Overfitting is even more likely when the number of candidate predictors is large relative to the effective sample size.
“Overfitting is even more likely when the number of candidate predictors is large relative to the effective sample size [10],[12],[70].”
From Reporting and Methods in Clinical Prediction Research: A Systematic Review (2012) · cites this paper as [70]
Apparent model performance is generally too optimistic, as the model has been tailored to the dataset at hand.
“This apparent model performance, however, is generally too optimistic, as the model has been tailored to the dataset at hand.”
From Reporting and Methods in Clinical Prediction Research: A Systematic Review (2012) · cites this paper as [70]

Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · see citing works
Ridge regression: Biased estimation for nonorthogonal problems
Hoerl, A. E., & Kennard, R. W. (1970) · Technometrics · doi:10.1080/00401706.1970.10488634 · Cited by 9,648 · see citing works
Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter
Golub, G. H., Heath, M. T., & Wahba, G. (1979) · Technometrics · doi:10.1080/00401706.1979.10489751 · Cited by 3,940 · see citing works
Theoretical Statistics
Cox, D. R., & Hinkley, D. V. (1974) · doi:10.1007/978-1-4899-2887-0 · Cited by 3,089 · see citing works
Ridge Regression: Applications to Nonorthogonal Problems
Hoerl, A. E., & Kennard, R. (1970) · Technometrics · doi:10.1080/00401706.1970.10488635 · Cited by 2,762 · see citing works
The Predictive Sample Reuse Method with Applications
Geisser, S. (1975) · Journal of the American Statistical Association · doi:10.1080/01621459.1975.10479865 · Cited by 2,748 · see citing works
Estimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300 · see citing works
Some Comments on C P
Mallows, C. L. (1973) · Technometrics · doi:10.2307/1267380 · Cited by 1,978 · see citing works
Goodness of fit tests for the multiple logistic regression model
Hosmer, D., & Lemesbow, S. (1980) · Communication in Statistics- Theory and Methods · doi:10.1080/03610928008827941 · Cited by 1,780 · see citing works
A Biometrics Invited Paper. The Analysis and Selection of Variables in Linear Regression
Hocking, R. R. (1976) · Biometrics · doi:10.2307/2529336 · Cited by 1,765 · see citing works
Estimation of Error Rates in Discriminant Analysis
Lachenbruch, P. A., & Mickey, M. R. (1968) · Technometrics · doi:10.1080/00401706.1968.10490530 · Cited by 1,508 · see citing works
The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction
Allen, D. M. (1974) · Technometrics · doi:10.1080/00401706.1974.10489157 · Cited by 1,469 · see citing works
Logistic Regression Diagnostics
Pregibon, D. (1981) · The Annals of Statistics · doi:10.1214/aos/1176345513 · Cited by 1,303 · see citing works
An asymptotic equivalence of choice of model by cross validation and Akaike's criterion
Stone, M. (1977) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1977.tb01603.x · Cited by 1,286 · see citing works
Regression, Prediction and Shrinkage
Copas, J. B. (1983) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1983.tb01258.x · Cited by 612 · see citing works
How Biased is the Apparent Error Rate of a Prediction Rule?
Efron, B. (1986) · Journal of the American Statistical Association · doi:10.2307/2289236 · Cited by 587 · see citing works
Mean Square Error of Prediction as a Criterion for Selecting Variables
Allen, D. M. (1971) · Technometrics · doi:10.1080/00401706.1971.10488811 · Cited by 562 · see citing works
An Optimal Selection of Regression Variables
Shibata, R. (1981) · Biometrika · doi:10.1093/biomet/68.1.45 · Cited by 562 · see citing works
Small Sample Properties of Robust Analyses of Linear Models Based on R-Estimates: A Survey
McKean, J. W., & Sheather, S. J. (1991) · The IMA volumes in mathematics and its applications · doi:10.1007/978-1-4612-4444-8_1 · Cited by 360 · see citing works
A note on the generalized information criterion for choice of a model
Atkinson, A. C. (1980) · Biometrika · doi:10.1093/biomet/67.2.413 · Cited by 260 · see citing works
Cross-Validation, the Jackknife, and the Bootstrap: Excess Error Estimation in Forward Logistic Regression
Gong, G. D. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478245 · Cited by 243 · see citing works
A Goodness-of-Fit Test for Binary Regression Models, Based on Smoothing Methods
le Cessie, S., & van Houwelingen, H. C. (1991) · Biometrics · doi:10.2307/2532385 · Cited by 221 · see citing works
Selection of Variables in Multiple Regression: Part I. A Review and Evaluation
Thompson, M. L. (1978) · International Statistical Review · doi:10.2307/1402505 · Cited by 196 · see citing works
Prediction of Future Observations in Growth Curve Models
Rao, C. R. (1987) · Statistical Science · doi:10.1214/ss/1177013119 · Cited by 175 · see citing works
How Many Variables Should Be Entered in a Regression Equation?
Breiman, L., & Freedman, D. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477941 · Cited by 158 · see citing works
Selection of Variables in Multiple Regression: Part II. Chosen Procedures, Computations and Examples
Thompson, M. L. (1978) · International Statistical Review · doi:10.2307/1402809 · Cited by 100 · see citing works
Predictability of the survival of patients with advanced ovarian cancer.
van Houwelingen, H. C., Huinink, W. W. T. B., van der Burg, M. E. L., et al. (1989) · Journal of Clinical Oncology · doi:10.1200/jco.1989.7.6.769 · Cited by 89 · see citing works
A Diagnostic for Cox Regression and General Conditional Likelihoods
Storer, B. E., & Crowley, J. W. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10477153 · Cited by 88 · see citing works
Discriminant Analysis and Clustering: Panel on Discriminant Analysis, Classification, and Clustering
(1989) · Statistical Science · doi:10.1214/ss/1177012666 · Cited by 83 · see citing works
The Use of Balanced Half-Sample Replication in Cross-Validation Studies
McCarthy, P. J. (1976) · Journal of the American Statistical Association · doi:10.1080/01621459.1976.10481534 · Cited by 58 · see citing works
Bootstrap and Cross-Validation Estimates of the Prediction Error for Linear Regression Models
Bunke, O., & Droge, B. (1984) · The Annals of Statistics · doi:10.1214/aos/1176346800 · Cited by 56 · see citing works
Comparison of the predictive power of different prognostic indices for overall survival in patients with advanced ovarian carcinoma.
Lund, B., Williamson, P., van Houwelingen, H. C., et al. (1990) · PubMed · Cited by 33 · see citing works
Logistic Regression, a review
van Houwelingen, H. C., & le Cessie, S. (1988) · Statistica Neerlandica · doi:10.1111/j.1467-9574.1988.tb01237.x · Cited by 26 · see citing works
Some Applications of Hierarchical Models in Kidney Transplantation
Gilks, W. R. (1987) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348505 · Cited by 13 · see citing works
On the Robustness of Shrinkage Predictors in Regression to Differences between Past and Future Data
Jones, M. C., & Copas, J. B. (1986) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1986.tb01405.x · Cited by 13 · see citing works

Flagged first, then by citations

Sorted by citations

Predictive habitat distribution models in ecology
Guisan, A., & Zimmermann, N. E. (2000) · Ecological Modelling · doi:10.1016/s0304-3800(00)00354-9 · Cited by 7,359 · see citing works
Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · see citing works
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746 · see citing works
Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist
Moons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018 · see citing works
Towards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928 · see citing works
Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes
Riley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409 · see citing works
External validation is necessary in prediction research:
Bleeker, S. E., Moll, H. A., Steyerberg, E. W., Donders, A. R. T., Derksen-Lubsen, G., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00207-5 · Cited by 710 · see citing works
Substantial effective sample sizes were required for external validation studies of predictive logistic regression models
Vergouwe, Y., Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.06.017 · Cited by 636 · see citing works
Internal and external validation of predictive models: A simulation study of bias and precision in small samples
Steyerberg, E. W., Bleeker, S. E., Moll, H. A., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00047-7 · Cited by 611 · see citing works
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600 · see citing works
Validation and updating of predictive logistic regression models: a study on sample size and shrinkage
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552 · see citing works
Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study
Van Calster, B., Van Hoorde, K., Valentin, L., Testa, A. C., Fischerova, D., Van Holsbeke, C., Savelli, L., Franchi, D., Epstein, E., Kaijser, J., et al. (2014) · BMJ · doi:10.1136/bmj.g5920 · Cited by 510 · see citing works
Use of Framingham risk score and new biomarkers to predict cardiovascular mortality in older people: population based observational cohort study
de Ruijter, W., Westendorp, R. G. J., Assendelft, W. J. J., den Elzen, W. P. J., de Craen, A. J. M., le Cessie, S., & Gussekloo, J. (2009) · BMJ · doi:10.1136/bmj.a3083 · Cited by 339 · see citing works
External Validity of Risk Models: Use of Benchmark Values to Disentangle a Case-Mix Effect From Incorrect Coefficients
Vergouwe, Y., Moons, K. G. M., & Steyerberg, E. W. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq223 · Cited by 243 · see citing works
Multivariable Modeling Strategies
Harrell, F. E. (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7_4 · Cited by 221 · see citing works
Augmented backward elimination: a pragmatic and purposeful way to develop statistical models
Dunkler, D., Plischke, M., Leffondré, K., & Heinze, G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0113677 · Cited by 194 · see citing works
Stability of clinical prediction models developed using statistical or machine learning methods
Riley, R. D., & Collins, G. S. (2023) · Biometrical Journal · doi:10.1002/bimj.202200302 · Cited by 211 · see citing works
Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small
Riley, R. D., Snell, K. I. E., Martin, G. P., Whittle, R., Archer, L., Sperrin, M., & Collins, G. S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.12.005 · Cited by 142 · see citing works
Application of Shrinkage Techniques in Logistic Regression Analysis: A Case Study
Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2001) · Statistica Neerlandica · doi:10.1111/1467-9574.00157 · Cited by 124 · see citing works
Regression shrinkage methods for clinical prediction models do not guarantee improved performance: Simulation study
Van Calster, B., van Smeden, M., De Cock, B., & Steyerberg, E. W. (2020) · Statistical Methods in Medical Research · doi:10.1177/0962280220921415 · Cited by 120 · see citing works
Survival Prediction Models since Liver Transplantation - Comparisons between Cox Models and Machine Learning Techniques
Kantidakis, G., Putter, H., Lancia, C., Boer, J. D., Braat, A. E., & Fiocco, M. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01153-1 · Cited by 75 · see citing works
Developing and Evaluating Prediction Models in Rehabilitation Populations
Seel, R. T., Steyerberg, E. W., Malec, J. F., Sherer, M., & Macciocchi, S. N. (2012) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2012.04.021 · Cited by 49 · see citing works
Development and validation of a prediction algorithm for use by health professionals in prediction of recurrence of major depression
Wang, J. L., Patten, S., Sareen, J., Bolton, J., Schmitz, N., & MacQueen, G. (2014) · Depression and Anxiety · doi:10.1002/da.22215 · Cited by 43 · see citing works
Ultrasound‐based risk model for preoperative prediction of lymph‐node metastases in women with endometrial cancer: model‐development study
Eriksson, L. S. E., Epstein, E., Testa, A. C., Fischerova, D., Valentin, L., Sladkevicius, P., Franchi, D., Frühauf, F., Fruscio, R., Haak, L. A., et al. (2020) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.21950 · Cited by 24 · see citing works
Covariate‐adjusted measures of discrimination for survival data
White, I. R., Rapsomaniki, E., & for the Emerging Risk Factors Collaboration (2015) · Biometrical Journal · doi:10.1002/bimj.201400061 · Cited by 13 · see citing works

Authors: Van Houwelingen, J. C., & Le Cessie, S.
Venue: Statistics in Medicine
Publisher: Wiley
Published: 1990-11
Volume: 9
Issue: 11
Pages: 1303-1325
ISSN: 0277-6715 (ISSN-L)
1097-0258
DOI: 10.1002/sim.4780091109
OpenAlex: W650692540
Cited by: 621
References: 35
Citations by year: 2026: 15 · 2025: 13 · 2024: 27 · 2023: 26 · 2022: 17 · 2021: 29 · 2020: 29 · 2019: 16 · 2018: 23 · 2017: 14 · 2016: 16 · 2015: 21 · 2014: 33 · 2013: 22 · 2012: 29
FWCI: 4.68
Type: journal article
Language: en
Open access: closed
Licence: onlinelibrary.wiley.com/termsAndConditions#vor (vor, 5824 d)
Locations: Statistics in Medicine (publishedVersion)
PubMed (publishedVersion)
Quality Engineering
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
Keywords: prediction error estimation, statistical models, cross-validation, ordinary least squares, logistic regression, Cox proportional hazards model, survival analysis
MeSH terms: Calibration, Female, Graft Survival, Humans, Ovarian Neoplasms, Prognosis, Regression Analysis, Models, Statistical, Least-Squares Analysis, Survival Analysis, Kidney Transplantation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Predictive value of statistical models.

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks