The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Pisters, R., Lane, D. A., Nieuwlaat, R., de Vos, C. B., Crijns, H. J. G. M., & Lip, G. Y. H. (2010) · Chest · doi:10.1378/chest.10-0134 · Cited by 4,505
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Towards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · cited 2×
“Besides discrimination, other metrics, such as calibration or reclassification, may also need to be reported in both derivation and validation studies [70,77,78].”
BackgroundPrognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436
Prognosis and prognostic research: Developing a prognostic model
Royston, P., Moons, K. G. M., Altman, D. G., & Vergouwe, Y. (2009) · BMJ · doi:10.1136/bmj.b604 · Cited by 1,172
Assessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160
Lip, G. Y. H., Frison, L., Halperin, J. L., & Lane, D. A. (2011) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2010.09.024 · Cited by 1,080
Hippisley-Cox, J., Coupland, C., Vinogradova, Y., Robson, J., May, M., & Brindle, P. (2007) · BMJ · doi:10.1136/bmj.39261.471806.55 · Cited by 1,038
Lemeshow, S. (1993) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1993.03510200084037 · Cited by 1,029
Prognosis and prognostic research: application and impact of prognostic models in clinical practice
Moons, K. G. M., Altman, D. G., Vergouwe, Y., & Royston, P. (2009) · BMJ · doi:10.1136/bmj.b606 · Cited by 934
Hasford, J., Pfirrmann, M., Hehlmann, R., Allan, N. C., Baccarani, M., Kluin-Nelemans, J. C., Alimena, G., Steegmann, J. L., & Ansari, H. (1998) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/90.11.850 · Cited by 819
Prognostic Indices for Older Adults: A Systematic Review
Yourman, L. C., Lee, S. J., Schonberg, M. A., Widera, E. W., & Smith, A. K. (2012) · JAMA · doi:10.1001/jama.2011.1966 · Cited by 736 · cited 1×
“Recently published systematic reviews of studies in prognostic research field have raised concerns about the proper validation and reporting strategies of new risk models [2,75,76].”
BackgroundExternal validation is necessary in prediction research:
Bleeker, S. E., Moll, H. A., Steyerberg, E. W., Donders, A. R. T., Derksen-Lubsen, G., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00207-5 · Cited by 710
Steyerberg, E. W., Bleeker, S. E., Moll, H. A., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00047-7 · Cited by 611
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600 · cited 1×
“Recently published systematic reviews of studies in prognostic research field have raised concerns about the proper validation and reporting strategies of new risk models [2,75,76].”
BackgroundGallamini, A. (2004) · Blood · doi:10.1182/blood-2003-09-3080 · Cited by 598
Yuen, M.-F., Tanaka, Y., Fong, D. Y.-T., Fung, J., Wong, D. K.-H., Yuen, J. C.-H., But, D. Y.-K., Chan, A. O.-O., Wong, B. C.-Y., Mizokami, M., et al. (2009) · Journal of Hepatology · doi:10.1016/j.jhep.2008.07.023 · Cited by 584
Risk models and scores for type 2 diabetes: systematic review
Noble, D., Mathur, R., Dent, T., Meads, C., & Greenhalgh, T. (2011) · BMJ · doi:10.1136/bmj.d7163 · Cited by 551 · cited 2×
“For example, there are 94 models to assess risk of incident diabetes in the medical literature, and only 14 of those have been calculated again in subsequent publications [2].”
BackgroundEvaluation of SOFA-based models for predicting mortality in the ICU: A systematic review
Minne, L., Abu-Hanna, A., & de Jonge, E. (2008) · Critical Care · doi:10.1186/cc7160 · Cited by 547
Collins, G. S., Mallett, S., Omar, O., & Yu, L.-M. (2011) · BMC Medicine · doi:10.1186/1741-7015-9-103 · Cited by 484
Wishart, G. C., Azzato, E. M., Greenberg, D. C., Rashbass, J., Kearins, O., Lawrence, G., Caldas, C., & Pharoah, P. D. (2010) · Breast Cancer Research · doi:10.1186/bcr2464 · Cited by 418
Successful Validation of the Palliative Prognostic Score in Terminally Ill Cancer Patients
Maltoni, M., Nanni, O., Pirovano, M., Scarpi, E., Indelli, M., Martini, C., Monti, M., Arnoldi, E., Piva, L., Ravaioli, A., et al. (1999) · Journal of Pain and Symptom Management · doi:10.1016/s0885-3924(98)00146-8 · Cited by 417
A New Palliative Prognostic Score
Pirovano, M., Maltoni, M., Nanni, O., Marinari, M., Indelli, M., Zaninetta, G., Petrella, V., Barni, S., Zecca, E., Scarpi, E., et al. (1999) · Journal of Pain and Symptom Management · doi:10.1016/s0885-3924(98)00145-6 · Cited by 413
Geersing, G.-J., Bouwmeester, W., Zuithoff, P., Spijker, R., Leeflang, M., & Moons, K. (2012) · PLoS ONE · doi:10.1371/journal.pone.0032844 · Cited by 350 · cited 1×
“There are standard searches for identifying the space of prognostic and diagnostic prediction studies in PubMed that have high sensitivity, that is, the Ingui and broad Haynes searches claim to have 98% and 96% sensitivity, respectively [93], and these can be very useful to apply in situations where searches are perfo…”
Systematic review of prognostic models in traumatic brain injury
Perel, P., Edwards, P., Wentz, R., & Roberts, I. (2006) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-6-38 · Cited by 344
Olsson, T., Terent, A., & Lind, L. (2004) · Journal of Internal Medicine · doi:10.1111/j.1365-2796.2004.01321.x · Cited by 323
Abbasi, A., Peelen, L. M., Corpeleijn, E., van der Schouw, Y. T., Stolk, R. P., Spijkerman, A. M. W., van der A, D. L., Moons, K. G. M., Navis, G., Bakker, S. J. L., et al. (2012) · BMJ · doi:10.1136/bmj.e5900 · Cited by 321
Comparisons of established risk prediction models for cardiovascular disease: systematic review
Siontis, G. C. M., Tzoulaki, I., Siontis, K. C., & Ioannidis, J. P. A. (2012) · BMJ · doi:10.1136/bmj.e3318 · Cited by 300 · cited 1×
“Comparisons of risk prediction models for cardiovascular disease have also shown allegiance or optimism bias: when the authors of the comparative study have developed one of the models, they report favorable results for their own model [10].”
BackgroundSelf-citationNolan, J. P., Laver, S. R., Welch, C. A., Harrison, D. A., Gupta, V., & Rowan, K. (2007) · Anaesthesia · doi:10.1111/j.1365-2044.2007.05232.x · Cited by 297
Ho, G. T., Mowat, C., Goddard, C. J. R., Fennell, J. M., Shah, N. B., Prescott, R. J., & Satsangi, J. (2004) · Alimentary Pharmacology & Therapeutics · doi:10.1111/j.1365-2036.2004.01945.x · Cited by 289
Proposal of a new prognostic model for hepatocellular carcinoma: an analysis of 403 patients
Tateishi, R., Yoshida, H., Shiina, S., Imamura, H., Hasegawa, K., Teratani, T., Obi, S., Sato, S., Koike, Y., Fujishima, T., et al. (2005) · Gut · doi:10.1136/gut.2003.035055 · Cited by 273
Ioannidis, J. P. A. (2011) · JAMA · doi:10.1001/jama.2011.713 · Cited by 252
Self-citationA new and feasible model for predicting operative risk
Donati, A., Ruzzi, M., Adrario, E., Pelaia, P., Coluzzi, F., Gabbanelli, V., & Pietropaoli, P. (2004) · British Journal of Anaesthesia · doi:10.1093/bja/aeh210 · Cited by 227
Hippisley-Cox, J., Coupland, C., Vinogradova, Y., Robson, J., & Brindle, P. (2008) · Heart · doi:10.1136/hrt.2007.134890 · Cited by 227
van Dieren, S., Beulens, J. W. J., Kengne, A. P., Peelen, L. M., Rutten, G. E. H. M., Woodward, M., van der Schouw, Y. T., & Moons, K. G. M. (2012) · Heart · doi:10.1136/heartjnl-2011-300734 · Cited by 216
for the oncoLR health network, Colinet, B., Jacot, W., Bertrand, D., Lacombe, S., Bozonnat, M.-C., Daurès, J.-P., & Pujol, J.-L. (2005) · British Journal of Cancer · doi:10.1038/sj.bjc.6602836 · Cited by 211
Harrison, D. A., Parry, G. J., Carpenter, J. R., Short, A., & Rowan, K. (2007) · Critical Care Medicine · doi:10.1097/01.ccm.0000259468.24532.44 · Cited by 209
Reporting methods in studies developing prognostic models in cancer: a review
Mallett, S., Royston, P., Dutton, S., Waters, R., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-20 · Cited by 206 · cited 1×
“Currently, numerous prediction models have been developed for a variety of clinical conditions, settings, and outcomes [2,82e86], but few such models have been properly derived, validated, reported in medical literature, and finally implemented in clinical practice [87,88].”
BackgroundWhy predictive indexes perform less well in validation studies. Is it magic or methods?
Charlson, M. E. (1987) · Archives of Internal Medicine · doi:10.1001/archinte.147.12.2155 · Cited by 205
Reporting performance of prognostic models in cancer: a review.
Mallett, S., Royston, P., Waters, R., Dutton, S., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-21 · Cited by 195
Wong, D. T., Cheng, D. C. H., Kustra, R., Tibshirani, R., Karski, J., Carroll-Munro, J., & Sandler, A. (1999) · Anesthesiology · doi:10.1097/00000542-199910000-00012 · Cited by 195
Röllig, C., Thiede, C., Gramatzki, M., Aulitzky, W., Bodenstein, H., Bornhäuser, M., Platzbecker, U., Stuhlmann, R., Schuler, U., Soucek, S., et al. (2010) · Blood · doi:10.1182/blood-2010-01-267302 · Cited by 176
Prediction models in reproductive medicine: a critical appraisal†
Leushuis, E., van der Steeg, J. W., Steures, P., Bossuyt, P. M. M., Eijkemans, M. J. C., van der Veen, F., Mol, B. W. J., & Hompes, P. G. A. (2009) · Human Reproduction Update · doi:10.1093/humupd/dmp013 · Cited by 173
Accuracy of Risk Scores for Patients With Chronic Hepatitis B Receiving Entecavir Treatment
Wong, G. L., Chan, H. L., Chan, H., Tse, P. C., Tse, Y., Mak, C. W., Lee, S. K., Ip, Z. M., Lam, A. T., Iu, H. W., et al. (2013) · Gastroenterology · doi:10.1053/j.gastro.2013.02.002 · Cited by 172
Predicting Death: An Empirical Evaluation of Predictive Tools for Mortality
Siontis, G. C. M., Tzoulaki, I., & Ioannidis, J. (2011) · Archives of Internal Medicine · doi:10.1001/archinternmed.2011.334 · Cited by 142
Self-citationSanchís, J. M., Bodí, V., Núñez, J. E., Bertomeu‐González, V., Gómez, C., Bosch, M. J., Consuegra, L., Bosch, X., Chorro, F. J., & Llácer, Á. (2005) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2005.04.037 · Cited by 140
Prognostic models for predicting mortality in elderly ICU patients: a systematic review
Minne, L., Ludikhuize, J., de Jonge, E., de Rooij, S. E., & Abu–Hanna, A. (2011) · Intensive Care Medicine · doi:10.1007/s00134-011-2265-6 · Cited by 138
Spelt, L., Andersson, B., Nilsson, J., & Andersson, R. (2011) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2011.10.013 · Cited by 136
A population-based validation of the prognostic model PREDICT for early breast cancer
Wishart, G. C., Bajdik, C. D., Azzato, E. M., Dicks, E., Greenberg, D. C., Rashbass, J., Caldas, C. H., & Pharoah, P. D. P. (2011) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2011.02.001 · Cited by 129
Jacot, W., Colinet, B., Bertrand, D., Lacombe, S., Bozonnat, M. C., Daurès, J., & Pujol, J.-L. (2008) · Annals of Oncology · doi:10.1093/annonc/mdn064 · Cited by 105
Smith, J. G., Wieloch, M., Koul, S. M., Braun, O. Ö., Lumsden, J., Rydell, E., Öhman, J. N., Scherstén, F., Svensson, P. J., & van der Pals, J. (2012) · EuroIntervention · doi:10.4244/eijv8i6a105 · Cited by 89
Minimal and Null Predictive Effects for the Most Popular Blood Biomarkers of Cardiovascular Disease
Ioannidis, J. P. A., & Tzoulaki, I. (2012) · Circulation Research · doi:10.1161/res.0b013e31824da8ad · Cited by 75
Self-citationYau, T. C. C., Yao, T. J., Chan, P., Ng, K. K., Fan, S. T., & Poon, R. T. P. (2008) · Cancer · doi:10.1002/cncr.23878 · Cited by 63
Scoring systems in acute pancreatitis: Which one to use in intensive care units?
Juneja, D., Gopal, P. B. N., & Ravula, M. (2010) · Journal of Critical Care · doi:10.1016/j.jcrc.2009.12.010 · Cited by 54
Wlodzimirow, K. A., Eslami, S., Chamuleau, R. A. F. M., Nieuwoudt, M., & Abu–Hanna, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0050952 · Cited by 49
Aceituno, M., García-Planella, E., Heredia, C., Zabana, Y., Feu, F., Domènech, E., Gassull, M. A., & Panés, J. (2007) · Inflammatory Bowel Diseases · doi:10.1002/ibd.20322 · Cited by 49
Comparison of questionnaire and quantitative ultrasound techniques as screening tools for DXA
Cook, R. B., Collins, D., Tucker, J., & Zioupos, P. (2005) · Osteoporosis International · doi:10.1007/s00198-005-1864-x · Cited by 49
Tassinari, D., Montanari, L., Maltoni, M. C., Ballardini, M., Piancastelli, A., Musi, M., Porzio, G., Minotti, V., Caraceni, A. T., Poggi, B., et al. (2007) · Supportive Care in Cancer · doi:10.1007/s00520-007-0302-3 · Cited by 48
Bonifazi, F., Vivo, A. D., Rosti, G., Tiribelli, M., Russo, D., Trabacchi, E., Fiacchini, M., Montefusco, E., & Baccarani, M. (2000) · British Journal of Haematology · doi:10.1111/j.1365-2141.2000.02366.x · Cited by 47
Model for predicting short-term mortality of severe sepsis
Adrie, C., Français, A., Alvarez-Gonzalez, A., Mounier, R., Azoulay, É., Zahar, J., Clec’h, C., Goldgran-Tolédano, D., Hammer, L., Descorps-Declere, A., et al. (2009) · Critical Care · doi:10.1186/cc7881 · Cited by 47
Limitations of risk score models in patients with acute chest pain
Manini, A. F., Dannemann, N., Brown, D., Butler, J., Bamberg, F., Nagurney, J. T., Nichols, J. H., & Hoffmann, U. (2008) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2008.01.022 · Cited by 39
Influence of fetal growth patterns on sonographic estimation of fetal weight
Simon, N. V., Levisky, J. S., Shearer, D. M., O'Lear, M. S., & Flood, J. T. (1987) · Journal of Clinical Ultrasound · doi:10.1002/jcu.1870150604 · Cited by 37
Rathore, S. S., Weinfurt, K. P., Foody, J. M., & Krumholz, H. M. (2005) · American Heart Journal · doi:10.1016/j.ahj.2005.03.069 · Cited by 36
Bonifazi, F., Vivo, A. D., Rosti, G., Tiribelli, M., Russo, D., Trabacchi, E., Fiacchini, M., Montefusco, E., & Baccarani, M. (2000) · British Journal of Haematology · doi:10.1046/j.1365-2141.2000.02366.x · Cited by 34
Kondo, K., Chijiiwa, K., Nagano, M., Hiyoshi, M., Kai, M., Maehara, N., Ohuchida, J., Nakao, H., & Ohkuwa, Y. (2007) · PubMed · Cited by 32
A new simple risk score in patients with acute chest pain without existing known coronary disease
Conti, A., Vanni, S., Taglia, B. D., Paladini, B., Magazzini, S., Grifoni, S., Nozzoli, C., & Gensini, G. F. (2010) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2008.10.023 · Cited by 25
Villella, M., Villella, A., Santoro, L., Santoro, E., Franzosi, M. G., & Maggioni, A. P. (2003) · American Heart Journal · doi:10.1067/mhj.2003.162 · Cited by 18
Ultrasonic tissue characterization of the placenta: Is it of clinical value?
Crawford, D. C., Fenton, D. W., & Price, W. I. (1985) · Journal of Clinical Ultrasound · doi:10.1002/1097-0096(199010)13:8<555::aid-jcu1870130807>3.0.co;2-z · Cited by 11
Conti, A., Poggioni, C., Viviani, G., Mariannini, Y., Luzzi, M., Cerini, G., Canuti, E., Zanobetti, M., Innocenti, F., & Pini, R. (2012) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2012.01.028 · Cited by 9
Valeur, N., Clemmensen, P. M., Grande, P., & Saunamäki, K. I. (2007) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2007.05.026 · Cited by 8
What do we mean by validating a prognostic model?
(2000) · doi:10.1002/(sici)1097-0258(20000229)19:4<453::aid-sim350>3.0.co;2-5
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4<361::aid-sim168>3.0.co;2-4
Cited by
Prediction models need appropriate internal, internal-external, and external validation
Steyerberg, E. W., & Harrell, F. E. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.005 · Cited by 1,304
Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936
Self-citationExternal validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887
Collins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544
Metadata
- Authors
- Siontis, G. C. M., Tzoulaki, I., Castaldi, P. J., & Ioannidis, J. P. A.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2015-01
- Volume
- 68
- Issue
- 1
- Pages
- 25-34
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2095987352
- Cited by
- 459
- References
- 74
- FWCI
- 18.22
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.