External validation of prognostic models for critically ill patients required substantial sample sizes

Peek, N.; Arts, D.G.T.; Bosman, R.J.; van der Voort, P.H.J.; de Keizer, N.F.

doi:10.1016/j.jclinepi.2006.08.011

Cited by 92Statistical Methods in EpidemiologySepsis Diagnosis and TreatmentMachine Learning in Healthcare

External validation of prognostic models for critically ill patients required substantial sample sizes

Peek, N., Arts, D.G.T., Bosman, R.J., van der Voort, P.H.J., de Keizer, N.F.·2007·Journal of Clinical Epidemiology

DOI 10.1016/j.jclinepi.2006.08.011

Open this paper in CoChat — free account

Abstract

Objective To investigate the behavior of predictive performance measures that are commonly used in external validation of prognostic models for outcome at intensive care units (ICUs).Study design and setting Four prognostic models (Simplified Acute Physiology Score II, the Acute Physiology and Chronic Health Evaluation II, and the Mortality Probability Models II) were evaluated in the Dutch National Intensive Care Evaluation registry database. For each model discrimination (AUC), accuracy (Brier score), and two calibration measures were assessed on data from 41,239 ICU admissions. This validation procedure was repeated with smaller subsamples randomly drawn from the database, and the results were compared with those obtained on the entire data set.Results Differences in performance between the models were small. The AUC and Brier score showed large variation with small samples. Standard errors of AUC values were accurate but the power to detect differences in performance was low. Calibration tests were extremely sensitive to sample size. Direct comparison of performance, without statistical analysis, was unreliable with either measure.Conclusion Substantial sample sizes are required for performance assessment and model comparison in external validation. Calibration statistics and significance tests should not be used in these settings. Instead, a simple customization method to repair lack-of-fit problems is recommended.

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2006.08.011

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Often, more data are needed when comparing calibration between models.
“Often, more data are needed when comparing calibration between models.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [101]

Applied logistic regression
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305 · see citing works
Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · see citing works
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
Applied Logistic Regression
Hosmer, D. W., & Lemeshow, S. (2000) · doi:10.1002/0471722146 · Cited by 20,302 · see citing works
APACHE II
Knaus, W. A., Draper, E. A., Wagner, D. P., & Zimmerman, J. E. (1985) · Critical Care Medicine · doi:10.1097/00003246-198510000-00009 · Cited by 15,333 · see citing works
APACHE II-A Severity of Disease Classification System
Knaus, W. A., Draper, E. A., Wagner, D. P., & Zimmerman, J. E. (1986) · Critical Care Medicine · doi:10.1097/00003246-198608000-00028 · Cited by 13,379 · see citing works
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270 · see citing works
A New Simplified Acute Physiology Score (SAPS II) Based on a European/North American Multicenter Study
Le Gall, J.-R. (1993) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.270.24.2957 · Cited by 7,773 · see citing works
Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878 · see citing works
Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis
Rao, S. J. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s263 · Cited by 1,986 · see citing works
Assessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160 · see citing works
Mortality Probability Models (MPM II) Based on an International Cohort of Intensive Care Unit Patients
Lemeshow, S. A. (1993) · JAMA · doi:10.1001/jama.1993.03510200084037 · Cited by 1,030 · see citing works
External validation is necessary in prediction research:
Bleeker, S. E., Moll, H. A., Steyerberg, E. W., Donders, A. R. T., Derksen-Lubsen, G., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00207-5 · Cited by 710 · see citing works
Internal and external validation of predictive models: A simulation study of bias and precision in small samples
Steyerberg, E. W., Bleeker, S. E., Moll, H. A., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00047-7 · Cited by 611 · see citing works
Two further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556 · see citing works
Validation and updating of predictive logistic regression models: a study on sample size and shrinkage
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552 · see citing works
Validation techniques for logistic regression models
Miller, M. E., Hui, S. L., & Tierney, W. M. (1991) · Statistics in Medicine · doi:10.1002/sim.4780100805 · Cited by 322 · see citing works
Evaluation of Acute Physiology and Chronic Health Evaluation III predictions of hospital mortality in an independent database
Zimmerman, J. E., Wagner, D. P., Draper, E. A., Wright, L., Alzola, C., & Knaus, W. A. (1998) · Critical Care Medicine · doi:10.1097/00003246-199808000-00012 · Cited by 261 · see citing works
Outcome prediction in intensive care: results of a prospective, multicentre, Portuguese study
Moreno, R. P., & Morais, P. (1997) · Intensive Care Medicine · doi:10.1007/s001340050313 · Cited by 160 · see citing works
ABC of intensive care: Outcome data and scoring systems
Gunning, K. E. J., & Rowan, K. (1999) · BMJ · doi:10.1136/bmj.319.7204.241 · Cited by 153 · see citing works
Factors affecting the performance of the models in the Mortality Probability Model II system and strategies of customization
Zhu, B., Lemeshow, S. A., Hosmer, D. W., Klar, J. M., Avrunin, J. S., & Teres, D. (1996) · Critical Care Medicine · doi:10.1097/00003246-199601000-00011 · Cited by 149 · see citing works
External validity of predictive models: a comparison of logistic regression, classification trees, and neural networks
Terrin, N. C., Schmid, C. H., Griffith, J. L., D’Agostino, R. B., & Selker, H. P. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00120-3 · Cited by 134 · see citing works
Quality of data collected for severity of illness scores in the Dutch National Intensive Care Evaluation (NICE) registry
Arts, D., de Keizer, N., Scheffer, G.-J., & de Jonge, E. (2002) · Intensive Care Medicine · doi:10.1007/s00134-002-1272-z · Cited by 134 · see citing works
Mortality Probability Models (MPM II) based on an international cohort of intensive care unit patients
Lemeshow, S. A. (1993) · JAMA · doi:10.1001/jama.270.20.2478 · Cited by 127 · see citing works
Tlie Measurement of Performance in Probabilistic Diagnosis
Hilden, J., Habbema, J. D. F., & Bjerregaard, B. E. (1978) · Methods of Information in Medicine · doi:10.1055/s-0038-1636443 · Cited by 126 · see citing works
Intensive Care Societyʼs Acute Physiology and Chronic Health Evaluation (APACHE II) study in Britain and Ireland
Rowan, K., Kerr, J., Major, E., McPherson, K., Short, A., & Vessey, M. P. (1994) · Critical Care Medicine · doi:10.1097/00003246-199409000-00007 · Cited by 124 · see citing works
Impact of different customization strategies in the performance of a general severity score
Moreno, R. P., & Apolone, G. (1997) · Critical Care Medicine · doi:10.1097/00003246-199712000-00017 · Cited by 94 · see citing works
Prognostic performance and customization of the SAPS II: results of a multicenter Austrian study
Metnitz, P., Valentin, A., Vesely, H., Alberti, C., Lang�, T., Lenz, K., Steltzer, H., & Hiesmayr, M. (1999) · Intensive Care Medicine · doi:10.1007/s001340050815 · Cited by 77 · see citing works
Simplified Acute Physiology Score II for measuring severity of illness in intermediate care units
Auriant, I., Vinatier, I., Thaler, F., Tourneur, M., & Loirat, P. (1998) · Critical Care Medicine · doi:10.1097/00003246-199808000-00023 · Cited by 75 · see citing works
Comparison of Acute Physiology and Chronic Health Evaluation II (APACHE II) and Simplified Acute Physiology Score II (SAPS II) scoring systems in a single Greek intensive care unit
Katsaragakis, S., Papadimitropoulos, K., Antonakis, P. T., Strergiopoulos, S., Konstadoulakis, M. M., & Androulakis, G. (2000) · Critical Care Medicine · doi:10.1097/00003246-200002000-00023 · Cited by 66 · see citing works
Training in data definitions improves quality of intensive care data
Arts, D. G. T., Bosman, R. J., de Jonge, E., Jc, J., & de Keizer, N. F. (2003) · Critical Care · doi:10.1186/cc1886 · Cited by 40 · see citing works
Predicting Outcome in the Intensive Care Unit Using Scoring Systems
Bertolini, G., D’Amico, R., Apolone, G., Cattaneo, A., Ravizza, A., Iapichino, G., Brazzi, L., & Melotti, R. M. (1998) · Medical Care · doi:10.1097/00005650-199809000-00009 · Cited by 27 · see citing works
National IC evaluation (NICE): a Dutch 1uality control system
de Keizer, N. F., & de Jonge, E. (2005) · Data Archiving and Networked Services (DANS) · Cited by 5 · see citing works
Assessing the potential for bias in meta-analysis due to selective reporting of subgroup analyses within studies
Hahn, S., Williamson, P. R., Hutton, J. L., Garner, P., & Flynn, E. V. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078
What do we mean by validating a prognostic model?
(2000) · doi:10.1002/(sici)1097-0258(20000229)19:4
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2

Flagged first, then by citations

Sorted by citations

Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878 · see citing works
Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist
Moons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018 · see citing works
External validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887 · see citing works
Sample size considerations for the external validation of a multivariable prognostic model: A resampling study
Collins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665 · see citing works
Validation, updating and impact of clinical prediction rules: A review
Toll, D. B., Janssen, K. J. M., Vergouwe, Y., & Moons, K. G. M. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.04.008 · Cited by 584 · see citing works
Association between administered oxygen, arterial partial oxygen pressure and mortality in mechanically ventilated intensive care unit patients
de Jonge, E., Peelen, L. M., Keijzers, P., Joore, H. C. A., de Lange, D. W., van der Voort, P. H., Bosman, R. J., de Waal, R. A. L., Wesselink, R. M. J., & de Keizer, N. F. (2008) · Critical Care · doi:10.1186/cc7150 · Cited by 425 · see citing works
Predicting outcomes in radiation oncology—multifactorial decision support systems
Lambin, P., van Stiphout, R. G. P. M., Starmans, M. H. W., Velazquez, E. R., Nalbantov, G. I., Aerts, H. J. W. L., Roelofs, E., van Elmpt, W., Boutros, P. C., Granone, P. M., et al. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.196 · Cited by 396 · see citing works
Assessing calibration of prognostic risk scores
Crowson, C. S., Atkinson, E. J., & Therneau, T. M. (2013) · Statistical Methods in Medical Research · doi:10.1177/0962280213497434 · Cited by 295 · see citing works
Predicting Acute Kidney Injury After Cardiac Surgery: A Systematic Review
Huen, S. C., & Parikh, C. R. (2012) · The Annals of Thoracic Surgery · doi:10.1016/j.athoracsur.2011.09.010 · Cited by 252 · see citing works
Multivariable Modeling Strategies
Harrell, F. E. (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7_4 · Cited by 221 · see citing works
Prognostic models for breast cancer: a systematic review
Phung, M. T., Tin, S. T., & Elwood, M. (2019) · BMC Cancer · doi:10.1186/s12885-019-5442-6 · Cited by 214 · see citing works
Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study
Riley, R. D., Snell, K. I. E., Archer, L., Ensor, J., Debray, T. P. A., CALSTER, B. V., van Smeden, M., & Collins, P. G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074821 · Cited by 213 · see citing works
Body Mass Index Is Associated With Hospital Mortality in Critically Ill Patients
Pickkers, P., de Keizer, N. F., Dusseljee, J., Weerheijm, D., van der Hoeven, J. G., & Peek, N. B. (2013) · Critical Care Medicine · doi:10.1097/ccm.0b013e31828a2aa1 · Cited by 191 · see citing works
Influence of secular trends and sample size on reference equations for lung function tests
Quanjer, P. H., Stocks, J., Cole, T. J., Hall, G. L., & Stanojevic, S. (2010) · European Respiratory Journal · doi:10.1183/09031936.00110010 · Cited by 180 · see citing works
Decision support systems for personalized and participative radiation oncology
Lambin, P., Zindler, J. D., Vanneste, B. G. L., Voorde, L. V. D., Eekers, D. B. P., Compter, I., Panth, K. M., Peerlings, J., Larue, R. T. H. M., Deist, T. M., et al. (2016) · Advanced Drug Delivery Reviews · doi:10.1016/j.addr.2016.01.006 · Cited by 154 · see citing works
Data Resource Profile: the Dutch National Intensive Care Evaluation (NICE) Registry of Admissions to Adult Intensive Care Units
van de Klundert, N., Holman, R., Dongelmans, D. A., & de Keizer, N. F. (2015) · International Journal of Epidemiology · doi:10.1093/ije/dyv291 · Cited by 151 · see citing works
How to derive and validate clinical prediction models for use in intensive care medicine
Labarère, J., Renaud, B., & Fine, M. J. (2014) · Intensive Care Medicine · doi:10.1007/s00134-014-3227-6 · Cited by 147 · see citing works
Validation of Clinical Scores Predicting Severe Acute Kidney Injury After Cardiac Surgery
Englberger, L., Suri, R. M., Li, Z., Dearani, J. A., Park, S. J., Sundt, T. M., & Schaff, H. V. (2010) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2010.04.017 · Cited by 122 · see citing works
The influence of volume and intensive care unit organization on hospital mortality in patients admitted with severe sepsis: a retrospective multicentre cohort study
Peelen, L. M., de Keizer, N. F., Peek, N. B., Scheffer, G. J., van der Voort, P. H. J., & de Jonge, E. (2007) · Critical Care · doi:10.1186/cc5727 · Cited by 106 · see citing works
Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works
Key steps and common pitfalls in developing and validating risk models
Wynants, L., Collins, P. G. S., & CALSTER, B. V. (2016) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.14170 · Cited by 95 · see citing works
Importance of sample size on the quality and utility of AI-based prediction models for healthcare
Riley, R. D., Ensor, J., Snell, K. I. E., Archer, L., Whittle, R., Dhiman, P., Alderman, J. E., Liu, X., Kirton, L., Manson-Whitton, J., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.01.013 · Cited by 93 · see citing works
Predicting Acute Renal Failure after Cardiac Surgery
Candela-Toha, Á. M., Accentn, E. E. A. A.-M. A., Abraira, V., Tenorio, M. A. A. T., Roux, D. P., de Pablo, A. A. A., Centella, T., & Tilde]o, F. L. (2008) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.00560208 · Cited by 84 · see citing works
Validation of four prognostic scores in patients with cancer admitted to Brazilian intensive care units: results from a prospective multicenter study
Soares, M., Silva, U. V. A., Teles, J. M. M., Silva, E., Caruso, P., Lobo, S. M., Dal‐Pizzol, F., Azevedo, L. C. P., de Carvalho, F. B., & Salluh, J. I. (2010) · Intensive Care Medicine · doi:10.1007/s00134-010-1807-7 · Cited by 74 · see citing works
Evaluation of simplified acute physiology score 3 performance: a systematic review of external validation studies
Nassar, A. P., Malbouisson, L. M. S., & Moreno, R. P. (2014) · Critical Care · doi:10.1186/cc13911 · Cited by 67 · see citing works
Assessing calibration of multinomial risk prediction models
Hoorde, K. V., Vergouwe, Y., Timmerman, D., Huffel, S. V., Steyerberg, E. W., & CALSTER, B. V. (2014) · Statistics in Medicine · doi:10.1002/sim.6114 · Cited by 65 · see citing works
Clinical prediction in defined populations: a simulation study investigating when and how to aggregate existing models
Martin, G. P., Mamas, M. A., Peek, N. B., Buchan, I. E., & Sperrin, M. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0277-1 · Cited by 60 · see citing works
Estimation of required sample size for external validation of risk models for binary outcomes
Pavlou, M., Chen, Q., Omar, R., Seaman, S. R., Steyerberg, E. W., White, I. R., & Ambler, G. (2021) · Statistical Methods in Medical Research · doi:10.1177/09622802211007522 · Cited by 58 · see citing works
Fracture Risk Assessment: State of the Art, Methodologically Unsound, or Poorly Reported?
Collins, G. S., & Michaëlsson, K. (2012) · Current Osteoporosis Reports · doi:10.1007/s11914-012-0108-1 · Cited by 51 · see citing works
Multicenter Validation of the max‐ICH Score in Intracerebral Hemorrhage
Sembill, J. A., Castello, J. P., Sprügel, M. I., Gerner, S. T., Hoelter, P., Lücking, H., Doerfler, A., Schwab, S., Huttner, H. B., Biffi, A., et al. (2020) · Annals of Neurology · doi:10.1002/ana.25969 · Cited by 49 · see citing works
Orphan G-protein coupled receptors: novel drug targets for the pharmaceutical industry.
Wilson, S., & Bergsma, D. J. (2000) · PubMed · Cited by 49 · see citing works
Cross-sectional and Longitudinal Spirometry in Children and Adolescents: Interpretative Strategies
Quanjer, P. H., Borsboom, G., Kivastik, J., Merkus, P. J. F. M., Hankinson, J., Houthuijs, D., Brunekreef, B., Ihorst, G., & Kühr, J. (2008) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.200805-700oc · Cited by 46 · see citing works
External validation of SAPS 3 and MPM0-III scores in 48,816 patients from 72 Brazilian ICUs
Moralez, G. M., Rabello, L. S. C. F., Lisboa, T. C., Lima, M. D. F. A., Hatum, R., Marco, F. V. C. D., Alves, A., Pinto, J. E. D. S. S., de Araújo, H. B. N., Ramos, G. V., et al. (2017) · Annals of Intensive Care · doi:10.1186/s13613-017-0276-3 · Cited by 46 · see citing works
Early Detection of Increased Intracranial Pressure Episodes in Traumatic Brain Injury: External Validation in an Adult and in a Pediatric Cohort
Güiza, F., Depreitere, B., Piper, I., Citerio, G., Jorens, P. G., Maas, A. I. R., Schuhmann, M. U., Lo, T.-Y. M., Donald, R., Jones, P. A., et al. (2016) · Critical Care Medicine · doi:10.1097/ccm.0000000000002080 · Cited by 44 · see citing works
Comparing Observed and Predicted Mortality Among ICUs Using Different Prognostic Systems
Kramer, A. A., Higgins, T. L., & Zimmerman, J. E. (2014) · Critical Care Medicine · doi:10.1097/ccm.0000000000000694 · Cited by 40 · see citing works
External validation of the paediatric logistic organ dysfunction score
Garcia, P. C. R., Eulmesekian, P. G., Branco, R. G., Pérez, A., Sfoggia, A., Olivero, L. J., Piva, J. P., & Tasker, R. C. (2009) · Intensive Care Medicine · doi:10.1007/s00134-009-1489-1 · Cited by 37 · see citing works
A Tree-Based Decision Model to Support Prediction of the Severity of Asthma Exacerbations in Children
Farion, K. J., Michalowski, W. J., Szymon Wilk, A., O’Sullivan, D. M., & Matwin, S. (2009) · Journal of Medical Systems · doi:10.1007/s10916-009-9268-7 · Cited by 35 · see citing works
SAPS 3 scores at the start of renal replacement therapy predict mortality in critically ill patients with acute kidney injury
Maccariello, E. R., Valente, C., Nogueira, L., Bonomo, H., Ismael, M., Machado, J. E., Baldotto, F., Godinho, M., Valença, R. V. R., Rocha, E., et al. (2009) · Kidney International · doi:10.1038/ki.2009.385 · Cited by 34 · see citing works
Predicting perioperative mortality after oesophagectomy: a systematic review of performance and methods of multivariate models
Warnell, I., Chincholkar, M., & Eccles, M. (2014) · British Journal of Anaesthesia · doi:10.1093/bja/aeu294 · Cited by 33 · see citing works
External validation and recalibration of the Brock model to predict probability of cancer in pulmonary nodules using NLST data
Winter, A., Aberle, D. R., & Hsu, W. W. Y. (2019) · Thorax · doi:10.1136/thoraxjnl-2018-212413 · Cited by 32 · see citing works
Abbreviated brief symptom inventory for use as an anxiety and depression screening instrument in primary care
Lang, A. J., Norman, S. B., Means‐Christensen, A. J., & Stein, M. B. (2008) · Depression and Anxiety · doi:10.1002/da.20471 · Cited by 32 · see citing works
Predicting ICU survival: A meta-level approach
Gortzis, L. G., Sakellaropoulos, F., Ilias, Ι., Stamoulis, K., & Dimopoulou, I. (2008) · BMC Health Services Research · doi:10.1186/1472-6963-8-157 · Cited by 31 · see citing works
Predictive performance of comorbidity measures in administrative databases for diabetes cohorts
Lix, L. M., Quail, J. M., Fadahunsi, O. O., & Teare, G. F. (2013) · BMC Health Services Research · doi:10.1186/1472-6963-13-340 · Cited by 30 · see citing works
Performance of Three Prognostic Models in Patients with Cancer in Need of Intensive Care in a Medical Center in China
Xing, X.-Z., Gao, Y., Wang, H., Huang, C., Qu, S.-N., Zhang, H., Wang, H., & Sun, K. (2015) · PLoS ONE · doi:10.1371/journal.pone.0131329 · Cited by 30 · see citing works
Using patient admission characteristics alone to predict mortality of critically ill patients: A comparison of 3 prognostic scores
Ho, K. M., Williams, T. A., Harahsheh, Y., & Higgins, T. L. (2015) · Journal of Critical Care · doi:10.1016/j.jcrc.2015.10.019 · Cited by 27 · see citing works
Severity of Illness and Predictive Models in Society of Critical Care Medicine’s First 50 Years: A Tale of Concord and Conflict
Kramer, A. A., Zimmerman, J. E., & Knaus, W. A. (2021) · Critical Care Medicine · doi:10.1097/ccm.0000000000004924 · Cited by 24 · see citing works
Predicting proliferative vitreoretinopathy: temporal and external validation of models based on genetic and clinical variables
Rojas, J., Fernández, I., Jimeno, J. C. P., MacLaren, R. E., Ramkissoon, Y., Harsum, S., Charteris, D. G., van Meurs, J. C., Amarakoon, S., García‐Arumí, J., et al. (2014) · British Journal of Ophthalmology · doi:10.1136/bjophthalmol-2014-305263 · Cited by 23 · see citing works
Ensuring comparisons of health-care providers are fair: development and validation of risk prediction models for critically ill patients
Harrison, D. A., Ferrando-Vivas, P., Shahin, J., & Rowan, K. (2015) · Health Services and Delivery Research · doi:10.3310/hsdr03410 · Cited by 20 · see citing works
Comparative Performance of Prediction Models for Contrast-Associated Acute Kidney Injury After Percutaneous Coronary Intervention
Ma, B., Allen, D. W., Graham, M. M., Har, B., Tyrrell, B. D., Tan, Z., Spertus, J. A., Brown, J. R., Matheny, M. E., Hemmelgarn, B. R., et al. (2019) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.119.005854 · Cited by 17 · see citing works
Severity of illness and outcome in ICU patients in the Netherlands: Results from the NICE registry 2006-2007
de Lange, D. W., Dusseljee, J., Brinkman, S., van Berkel, G., van Maanen, R., Bosman, R.-J., Joore, H. C. A., de Keizer, N. F., van de Voort, P. H. J., de Waal, R., et al. (2009) · Data Archiving and Networked Services (DANS) · Cited by 16 · see citing works
Severity of illness and organ dysfunction scoring systems in pediatric critical care: The impacts on clinician's practices and the future
Recher, M., Leteurtre, S., Canon, V., Baudelet, J., Lockhart-Bouron, M., & Hubert, H. (2022) · Frontiers in Pediatrics · doi:10.3389/fped.2022.1054452 · Cited by 15 · see citing works
Prediction of in-hospital bleeding in acutely ill medical patients: External validation of the IMPROVE bleeding risk score
Villiger, R., Julliard, P., Farhoumand, P. D., Choffat, D., Tritschler, T., Stalder, O., Rossel, J., Aujesky, D., Méan, M., & Baumgartner, C. (2023) · Thrombosis Research · doi:10.1016/j.thromres.2023.08.003 · Cited by 13 · see citing works
The Performance of Customised APACHE II and SAPS II in Predicting Mortality of Mixed Critically Ill Patients in a Thai Medical Intensive Care Unit
Khwannimit, B., & Bhurayanontachai, R. (2009) · Anaesthesia and Intensive Care · doi:10.1177/0310057x0903700515 · Cited by 13 · see citing works
Pediatric Trauma Assessment and Management Database: Leveraging Existing Data Systems to Predict Mortality and Functional Status after Pediatric Injury
Flynn‐O’Brien, K. T., Fallat, M. E., Rice, T. B., Gall, C. M., Nance, M. L., Upperman, J. S., Gourlay, D. M., Crow, J. P., & Rivara, F. P. (2017) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2017.01.061 · Cited by 12 · see citing works
Head-to-head comparison of 14 prediction models for postoperative delirium in elderly non-ICU patients: an external validation study
Wong, C. K., Munster, B. C. V., Hatseras, A., Veld, E. H. I. '., van Leeuwen, B. L., de Rooij, S. E., & Pleijhuis, R. G. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-054023 · Cited by 12 · see citing works
Desfecho de pacientes com câncer internados em unidades de terapia intensiva brasileiras com lesão renal aguda
Soares, M., Lobo, S. M., Torelly, A. P., Mello, P. M. V. D. C., Silva, U. V. A., Teles, J. M. M., Silva, E., Caruso, P., Friedman, G., Souza, P. C. P. D. S. E., et al. (2010) · Revista Brasileira de Terapia Intensiva · doi:10.1590/s0103-507x2010000300004 · Cited by 9 · see citing works
Prognostic models for locally advanced cervical cancer: external validation of the published models
Lora, D., de la Cámara, A. G., Fernández, S. P., de Salamanca, R. E., & Gómez, J. F. P.-R. (2017) · Journal of Gynecologic Oncology · doi:10.3802/jgo.2017.28.e58 · Cited by 9 · see citing works
Prognostic Calculator of the Clinical Response to Antihistamines in Chronic Urticaria: External Validation
Sánchez, J., Caraballo, A., Chérrez-Ojeda, I., García, E., Larco, J. I., Ramon, G. D., Velásquez‐Lopera, M. M., & Jaimes, F. A. (2025) · The Journal of Allergy and Clinical Immunology In Practice · doi:10.1016/j.jaip.2025.05.051 · Cited by 7 · see citing works
External validation of the European Society of Thoracic Surgeons morbidity and mortality risk models
de Antonio, D. G., Carrasco, S. C., Román, A. R., Royuela, A., Barturen, M. G., Obiols, C., Call, S., Royo, Í., Díaz, J. L. R., Cabañero-Sánchez, A., et al. (2022) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezac170 · Cited by 7 · see citing works
Prediction of very early major bleeding risk in acute pulmonary embolism: an independent external validation of the Pulmonary Embolism-Syncope, Anemia, and Renal Dysfunction (PE-SARD) bleeding score
Villiger, R., Méan, M., Stalder, O., Limacher, A., Rodondi, N., Righini, M., & Aujesky, D. (2023) · Journal of Thrombosis and Haemostasis · doi:10.1016/j.jtha.2023.04.025 · Cited by 6 · see citing works
External validation of the sepsis severity score
Wełna, M., Adamik, B., Goździk, W., & Kübler, A. (2020) · International Journal of Immunopathology and Pharmacology · doi:10.1177/2058738420936386 · Cited by 6 · see citing works
The Residual-Based Predictiveness Curve: A Visual Tool to Assess the Performance of Prediction Models
Casalicchio, G., Bischl, B., Boulesteix, A., & Schmid, M. C. (2015) · Biometrics · doi:10.1111/biom.12455 · Cited by 6 · see citing works
External validation of two diabetes risk scores in a young UK South Asian population
Gray, L. J., Khunti, K. K., Wilmot, E. G., Yates, T. E., & Davies, M. J. (2014) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2014.03.018 · Cited by 6 · see citing works
External Validation of Carbapenem-Resistant Enterobacteriaceae Acquisition Risk Prediction Model in a Medium Sized Hospital
Seo, S. M., & Jeong, I. S. (2020) · Journal of Korean Academy of Nursing · doi:10.4040/jkan.20137 · Cited by 6 · see citing works
GAN-Augmented Naïve Bayes for identifying high-risk coronary artery disease patients using CT angiography data
Zhang, L., Haldorai, A., & Naik, N. (2024) · Scientific Reports · doi:10.1038/s41598-024-73176-3 · Cited by 5 · see citing works
Comparative performance and external validation of three different scores in predicting inadequate bowel preparation among Greek inpatients undergoing colonoscopy
Gkolfakis, P. (2022) · Annals of Gastroenterology · doi:10.20524/aog.2022.0757 · Cited by 5 · see citing works
A validation study of the CirCom comorbidity score in an English cirrhosis population using the Clinical Practice Research Datalink
Crooks, C. J., West, J., & Jepsen, P. (2018) · Clinical Epidemiology · doi:10.2147/clep.s147535 · Cited by 5 · see citing works
External validation and calibration of the HoLISTIC Consortium's advanced‐stage Hodgkin lymphoma international prognostic index (A‐ HIPI ) in the Brazilian Hodgkin lymphoma registry
Buccheri, V., Moreira, F. R., Biasoli, I., de Castro, N. S., Villarim, C. C., Traina, F., da Silveira, T. M. B., Praxedes, M. K., Solza, C., Perobelli, L., et al. (2024) · British Journal of Haematology · doi:10.1111/bjh.19824 · Cited by 4 · see citing works
Prediction of Inhospital Mortality in Critically Ill Patients With Sepsis: Confirmation of the Added Value of 24-Hour Lactate to Acute Physiology and Chronic Health Evaluation IV
Baysan, M., Arbous, M. S., Steyerberg, E. W., & van der Bom, J. G. (2022) · Critical Care Explorations · doi:10.1097/cce.0000000000000750 · Cited by 5 · see citing works
Model for End-Stage Liver Disease and Seven Derivations to Prioritize Liver Transplant Candidates: Which Is the Winner?
Tohidinezhad, F., Eslami, S., Abu–Hanna, A., & Aliakbarian, M. (2018) · Experimental and Clinical Transplantation · doi:10.6002/ect.2017.0149 · Cited by 3 · see citing works
Calibration of Prognostic Risk Scores
CALSTER, B. V., & Steyerberg, E. W. (2018) · Wiley eBooks · doi:10.1002/9781118445112.stat08078 · Cited by 3 · see citing works
An integrated decision tree-artificial neural network hybrid to estimate clinical outcomes : ICU mortality and pre-term birth
Yu, N. (2010) · doi:10.22215/etd/2010-09185 · Cited by 3 · see citing works
External Validation of Models for Predicting Pneumonia after Cardiac Surgery
Santos, M., Tura, B. R., Rouge, A., & Braga, J. U. (2011) · Surgical Infections · doi:10.1089/sur.2010.014 · Cited by 2 · see citing works
Les scores de gravité généraux et de dysfonctions d’organes en réanimation pédiatrique : quoi de neuf en 2016 ?
Leteurtre, S., Lampin, M. E., Grandbastien, B., Recher, M., & Duhamel, A. (2016) · Médecine Intensive Réanimation · doi:10.1007/s13546-016-1220-5 · Cited by 2 · see citing works
Validation of age, creatinine and ejection fraction (ACEF) and Cockcroft-Gault ACEF scores in elective cardiac surgery.
Borracci, R. A., Rubio, M., Baldi, J., Ingino, C. A., & Barisani, J. L. (2017) · PubMed · Cited by 2 · see citing works
Excluding deep vein thrombosis in primary care: validation, updating, and implementation of a diagnostic rule.
Toll, D. B. (2008) · Utrecht University Repository (Utrecht University) · Cited by 2 · see citing works
Improvements in clinical prediction research
Janssen, K. J. M. (2007) · Utrecht University Repository (Utrecht University) · Cited by 1 · see citing works
Editorial Comment to J apan C ancer of the P rostate R isk A ssessment for combined androgen blockade including bicalutamide: Clinical application and validation
Yoshida, T. (2012) · International Journal of Urology · doi:10.1111/iju.12044 · Cited by 1 · see citing works
Multifactorial decision support systems in radiation oncology : clinical predictors and radiomics
Velazquez, E. R. (2014) · doi:10.26481/dis.20141024er · Cited by 1 · see citing works
Predictive modeling of treatment outcome in rectal cancer
van Stiphout, R. G. P. M. (2014) · doi:10.26481/dis.20140410rs
Reply to Collins and Le Manach
Kieser, T. M., Rose, M. S., & Head, S. J. (2016) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezw263
ORIGINAL INVESTIGATIONS Pathogenesis and Treatment of Kidney Disease Validation of Clinical Scores Predicting Severe Acute Kidney Injury After Cardiac Surgery
Englberger, L., Suri, R. M., Li, Z., Dearani, J. A., Park, S. J., Sundt, T. M., & Schaff, H. V. (2010)
COMPARATIVE PERFORMANCE AND EXTERNAL VALIDATION OF THREE DIFFERENT SCORES IN PREDICTING INADEQUATE BOWEL PREPARATION AMONG INPATIENTS UNDERGOING COLONOSCOPY
Gkolfakis, P., Kapizioni, C., Tziatzios, G., Facciorusso, A., Frazzoni, L., Thomopoulos, K., Potamianos, S. P., Christodoulou, D. K., Papadopoulos, V. N., Fuccio, L., et al. (2022) · Endoscopy · doi:10.1055/s-0042-1745055
Reply
Sgourakis, G. G., & Papapanagiotou, A. (2019) · Pancreas · doi:10.1097/mpa.0000000000001219
Comparação da performance preditiva de escores prognósticos em pacientes vítimas de trauma internados em unidade de terapia intensiva: um estudo de validação externa de SAPS 3, SOFA, ISS e NISS
Roepke, R. M. L. (2023) · doi:10.11606/t.5.2023.tde-23062023-163039
Reply to Collis et al.
Rauseo, A. M., & Spec, A. (2023) · Open Forum Infectious Diseases · doi:10.1093/ofid/ofad322
Validación externa y actualización de un modelo predictivo de lesión renal aguda en pacientes adultos hospitalizados en cuidados intensivos
Rincón-Castellanos, S. P., Barrios-Meza, J. L., & Rojas‐Gualdrón, D. F. (2023) · Acta Colombiana de Cuidado Intensivo · doi:10.1016/j.acci.2023.07.008
Real-Time Clinical Decision Support Systems for the Prevention of Deterioration in Acute Care
Blythe, R. D. (2024) · Queensland University of Technology · doi:10.5204/thesis.eprints.251291
Severity of Illness and Organ Dysfunction Scoring in Pediatric Intensive Care Units: Developments in 2020
Leteurtre, S., Recher, M., & Delangue, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-64762-8_3
Models for Categorical and Count Outcomes
Khan, S., & Rahman, M. S. (2025) · Springer eBooks · doi:10.1007/978-981-96-0878-2_12
Externally validated risk prediction models for gestational diabetes mellitus: A systematic review and meta‐analysis
Seifu, B. L., Belsti, Y., Yimer, N. B., Goldstein, R. F., Enticott, J., & Teede, H. J. (2026) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1111/aogs.70300

Predicting COVID-19 prognosis in the ICU remained challenging: external validation in a multinational regional cohort
Meijs, D. A. M., van Kuijk, S. M. J., Wynants, L., Stessel, B., Mehagnoul-Schipper, J., Hana, A., Scheeren, C. I. E., Bergmans, D. C. J. J., Bickenbach, J., Laenen, M. V., et al. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.10.015 · Cited by 14 · see citing works
External validation of the SAPS II, APACHE II and APACHE III prognostic models in South England: a multicentre study
Beck, D. H., Smith, G. B., Pappachan, J., & Millar, B. (2003) · Intensive Care Medicine · doi:10.1007/s00134-002-1607-9 · Cited by 149 · see citing works
Mortality prediction models in the adult critically ill: A scoping review
Keuning, B. E., Kaufmann, T., Wiersema, R., Granholm, A., Pettilä, V., Møller, M. H., Christiansen, C. F., Forte, J. C., Snieder, H., Keus, F., et al. (2019) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.13527 · Cited by 71 · see citing works
External Validation of Mortality Prediction Models for Critical Illness Reveals Preserved Discrimination but Poor Calibration
Cox, E. G. M., Wiersema, R., Eck, R. J., Kaufmann, T., Granholm, A., Vaara, S. T., Møller, M. H., van Bussel, B. C. T., Snieder, H., Pleijhuis, R. G., et al. (2022) · Critical Care Medicine · doi:10.1097/ccm.0000000000005712 · Cited by 26 · see citing works
Sample size considerations for the external validation of a multivariable prognostic model: A resampling study
Collins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665 · see citing works

Authors: Peek, N., Arts, D. G. T., Bosman, R. J., van der Voort, P. H. J., & de Keizer, N. F.
Affiliations: Amsterdam UMC Location University of Amsterdam (NL)
OLVG (NL)
Corresponding author: Peek, N.
ORCID: Peek, N.
Venue: Journal of Clinical Epidemiology
Publisher: Elsevier BV
Published: 2007-05
Volume: 60
Issue: 5
Pages: 491.e1-491.e13
ISSN: 0895-4356 (ISSN-L)
DOI: 10.1016/j.jclinepi.2006.08.011
OpenAlex: W1997812541
Cited by: 92
References: 35
Citations by year: 2026: 1 · 2025: 5 · 2024: 4 · 2023: 5 · 2022: 6 · 2021: 2 · 2020: 4 · 2019: 4 · 2018: 3 · 2017: 5 · 2016: 5 · 2015: 8 · 2014: 10 · 2013: 3 · 2012: 4
FWCI: 6.63
Type: journal article
Language: en
Open access: closed
Licence: www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations: Journal of Clinical Epidemiology (publishedVersion)
PubMed (publishedVersion)
Pure Amsterdam UMC (publishedVersion)
Data Archiving and Networked Services (DANS) (submittedVersion)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords: external validation, prognostic model, intensive care unit, sample size, model discrimination, calibration, AUC, Brier score, APACHE II score, SAPS II, ICU mortality prediction, lack of fit, model comparison
MeSH terms: Aged, Calibration, Epidemiologic Methods, Female, Humans, Intensive Care Units, Male, Middle Aged, Netherlands, Prognosis, Critical Illness, Outcome Assessment, Health Care

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

External validation of prognostic models for critically ill patients required substantial sample sizes

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks