A guide to systematic review and meta-analysis of prediction model performance
Open accessAbstract
Validation of prediction models is highly recommended and increasingly common in the literature. A systematic review of validation studies is therefore helpful, with meta-analysis needed to summarise the predictive performance of the model being validated across different settings and populations. This article provides guidance for researchers systematically reviewing and meta-analysing the existing evidence on a specific prediction model, discusses good practice when quantitatively summarising the predictive performance of the model across studies, and provides recommendations for interpreting meta-analysis estimates of model performance. We present key steps of the meta-analysis and illustrate each step in an example review, by summarising the discrimination and calibration performance of the EuroSCORE for predicting operative mortality in patients undergoing coronary artery bypass grafting.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previously discussed meta-analysis methods mainly focused on diagnostic prediction models with a binary outcome and may have limited value for prognostic models with a time-to-event outcome.
“These methods mainly focused on (diagnostic) prediction models with a binary outcome, and may therefore have limited value when reviewing (prognostic) models with a time-to-event outcome.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [14]
The total O:E ratio quantifies the averaged observed:expected ratio across the entire range of predicted risks and is strongly related to calibration-in-the-large.
“In particular, the total O:E ratio quantifies the averaged observed:expected ratio across the entire range of predicted risks, and is strongly related to calibration-in-the-large.14”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [14]
Discrimination and calibration performance are highly dependent on patient spectrum (case-mix variation) and therefore most likely to vary across validation studies.
“In particular, discrimination and calibration performance are highly dependent on patient spectrum (case-mix variation) and therefore most likely to vary across validation studies.14”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [14]
It is common for validation studies to present information on the total number of observed and expected events, or the corresponding observed frequencies, which can be used to calculate the total O:E ratio.
“However, it is common for validation studies to present information on the total number of observed (O) and expected (E) events, or the corresponding observed frequencies PO and PE. These quantities can then be used to calculate the total O:E ratio”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [14]
Quantitative synthesis of prediction model studies may help to better understand their potential generalizability and can be achieved by applying meta-analysis methods.
“Quantitative synthesis of prediction model studies may help to better understand their potential generalizability and can be achieved by applying meta-analysis methods.14”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [14]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Two new Later Stone Age sites from the Final Pleistocene in the Falémé Valley, eastern Senegal
Ndiaye, M., Lespez, L., Tribolo, C., Rassé, M., Hadjas, I., Davidoux, S., Huysecom, É., & Douze, K. (2024) · Monash University Research Portal (Monash University) · doi:10.3929/ethz-b-000667478 · Cited by 24,015 · see citing works
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648 · see citing works
Estimating the mean and variance from the median, range, and the size of a sample
Hozo, S. P., Djulbegovic, B., & Hozo, I. (2005) · BMC Medical Research Methodology · doi:10.1186/1471-2288-5-13 · Cited by 9,062 · see citing works
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267 · see citing works
Self-citationInterpretation of random effects meta-analyses
Riley, R. D., Higgins, J. P. T., & Deeks, J. J. (2011) · BMJ · doi:10.1136/bmj.d549 · Cited by 2,884 · see citing works
Self-citationChapter 9: Analysing Data and Undertaking Meta-Analyses
Deeks, J. J., Higgins, J. P. T., & Altman, D. G. (2008) · Cited by 2,782 · see citing works
IntHout, J., Ioannidis, J. P., & Borm, G. F. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-25 · Cited by 2,419 · see citing works
Stewart, L. A., Clarke, M., Rovers, M., Riley, R. D., Simmonds, M., Stewart, G., & Tierney, J. F. (2015) · JAMA · doi:10.1001/jama.2015.3656 · Cited by 2,134 · see citing works
Self-citationMoons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018 · see citing works
Self-citationAnalysing Data and Undertaking Meta‐Analyses
Deeks, J. J., Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch9 · Cited by 1,800 · see citing works
Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · see citing works
Self-citationVickers, A. J., Van Calster, B., & Steyerberg, E. W. (2016) · BMJ · doi:10.1136/bmj.i6 · Cited by 1,163 · see citing works
Assessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160 · see citing works
Iorio, A., Spencer, F. A., Falavigna, M., Alba, C., Lang, E., Burnand, B., McGinn, T., Hayden, J., Williams, K., Shea, B., et al. (2015) · BMJ · doi:10.1136/bmj.h870 · Cited by 880 · see citing works
Collins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729 · see citing works
Self-citationBayesian methods in meta-analysis and evidence synthesis
Sutton, A. J., & Abrams, K. R. (2001) · Statistical Methods in Medical Research · doi:10.1177/096228020101000404 · Cited by 712 · see citing works
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600 · cited 1× · see citing works
“12 27 Therefore, we propose to extract information on the total number of observed (O) and expected (E) events, which are statistics most likely to be reported or derivable (appendix 7).”
BackgroundSelf-citationDebray, T. P. A., Vergouwe, Y., Koffijberg, H., Nieboer, D., Steyerberg, E. W., & Moons, K. G. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.06.018 · Cited by 597 · see citing works
Self-citationRiley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585 · see citing works
Self-citationBerlin, J. A., Santanna, J., Schmid, C. H., Szczech, L. A. M., & Feldman, H. I. (2002) · Statistics in Medicine · doi:10.1002/sim.1023 · Cited by 468 · see citing works
Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change
Cornell, J. E., Mulrow, C. D., Localio, R., Stack, C. B., Meibohm, A. R., Guallar, E., & Goodman, S. N. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2886 · Cited by 453 · see citing works
Geersing, G.-J., Bouwmeester, W., Zuithoff, P., Spijker, R., Leeflang, M., & Moons, K. (2012) · PLoS ONE · doi:10.1371/journal.pone.0032844 · Cited by 350 · see citing works
Self-citationSystematic review of prognostic models in traumatic brain injury
Perel, P., Edwards, P. J., Wentz, R., & Roberts, I. (2006) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-6-38 · Cited by 343 · see citing works
Assessment of Claims of Improved Prediction Beyond the Framingham Risk Score
Tzoulaki, I. (2009) · JAMA · doi:10.1001/jama.2009.1757 · Cited by 276 · see citing works
Vergouwe, Y., Moons, K. G. M., & Steyerberg, E. W. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq223 · Cited by 243 · see citing works
Self-citationBayesian methods in meta-analysis and evidence synthesis
Sutton, A. J., & Abrams, K. R. (2001) · Statistical Methods in Medical Research · doi:10.1191/096228001678227794 · Cited by 216 · see citing works
Prediction of first coronary events with the Framingham score: A systematic review
Eichler, K., Puhan, M. A., Steurer, J., & Bachmann, L. M. (2007) · American Heart Journal · doi:10.1016/j.ahj.2007.02.027 · Cited by 214 · see citing works
Collins, G. S., Omar, O., Shanyinde, M., & Yu, L.-M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.06.020 · Cited by 192 · see citing works
Searching for Clinical Prediction Rules in MEDLINE
Ingui, B. J., & Rogers, M. A. M. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080391 · Cited by 184 · see citing works
Risk Models to Predict Hypertension: A Systematic Review
Echouffo-Tcheugui, J. B., Batty, G. D., Mika Kivimaki, B., & Kengne, A. P. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067370 · Cited by 152 · see citing works
Peat, G., Riley, R. D., Croft, P. R., Morley, K. I., Kyzas, P., Moons, K. G. M., Perel, P., Steyerberg, E. W., Schroter, S., Altman, D. G., et al. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001671 · Cited by 152 · see citing works
Self-citationPerformance of the original EuroSCORE
Siregar, S., Groenwold, R. H. H., de Heer, F., Bots, M. L., van der Graaf, Y., & van Herwerden, L. A. (2012) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezr285 · Cited by 132 · see citing works
Debray, T. P. A., Riley, R. D., Rovers, M. M., Reitsma, J. B., Moons, K. G. M., & Cochrane IPD Meta-analysis Methods group (2015) · PLoS Medicine · doi:10.1371/journal.pmed.1001886 · Cited by 130 · see citing works
Self-citationQin, G., & Hotilovac, L. (2007) · Statistical Methods in Medical Research · doi:10.1177/0962280207087173 · Cited by 126 · see citing works
Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE.
Wong, S. S.-L., Wilczynski, N. L., Haynes, R. B., & Ramkissoonsingh, R. (2003) · PubMed · Cited by 102 · see citing works
Blanche, P., Dartigues, J., & Jacqmin‐Gadda, H. (2013) · Biometrical Journal · doi:10.1002/bimj.201200045 · Cited by 94 · see citing works
A Prescription for Improving Drug Formulary Decision Making
Schiff, G. D., Galanter, W., Duhig, J., Koronkowski, M. J., Lodolce, A. E., Pontikes, P., Busker, J., Touchette, D. R., Walton, S. M., & Lambert, B. L. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001220 · Cited by 87 · see citing works
Straus, S. E., & Rath, D. (2004) · ACP Journal Club · doi:10.7326/acpjc-2004-140-1-a15 · Cited by 79 · see citing works
Snell, K. I. E., Hua, H., Debray, T. P. A., Ensor, J., Look, M. P., Moons, K. G. M., & Riley, R. D. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.05.009 · Cited by 75 · see citing works
Self-citationAssessing Risk Prediction Models Using Individual Participant Data From Multiple Studies
Pennells, L. A. M., Kaptoge, S. K., White, I. R., Thompson, S. G., Wood, A., Tipping, R., Folsom, A. R., Couper, D. J., Ballantyne, C. M., Coresh, J., et al. (2013) · American Journal of Epidemiology · doi:10.1093/aje/kwt298 · Cited by 63 · see citing works
Assessing discriminative ability of risk models in clustered data
Klaveren, D. V., Steyerberg, E. W., Perel, P., & Vergouwe, Y. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-5 · Cited by 61 · see citing works
A new concordance measure for risk prediction models in external validation settings
van Klaveren, D., Gönen, M., Steyerberg, E. W., & Vergouwe, Y. (2016) · Statistics in Medicine · doi:10.1002/sim.6997 · Cited by 48 · see citing works
Austin, P. C., Pencinca, M. J., & Steyerberg, E. W. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280214567141 · Cited by 39 · see citing works
Jinks, R. C., Royston, P., & Parmar, M. K. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0078-y · Cited by 32 · see citing works
Khan, K. S., Kunz, R., Kleijnen, J., & Antes, G. (2003) · Cited by 19 · see citing works
Ban, J.-W., Emparanza, J. I., Urreta, I., & Burls, A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0145779 · Cited by 18 · see citing works
Development and application of statistical methods for prognosis research
Snell, K. I. (2015) · University of Birmingham Institutional Research Archive (University of Birmingham) · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group† (2019) · Annals of Internal Medicine · doi:10.7326/m18-1376 · Cited by 2,984 · see citing works
Self-citationCalculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · see citing works
Self-citationMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669 · see citing works
Self-citationRiley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409 · see citing works
Self-citationMoons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-082505 · Cited by 1,069 · see citing works
Self-citationA guide to systematic review and meta-analysis of prognostic factor studies
Riley, R. D., Moons, K. G. M., Snell, K. I. E., Ensor, J., Hooft, L., Altman, D. G., Hayden, J. A., Collins, P. G. S., & Debray, T. P. A. (2019) · BMJ · doi:10.1136/bmj.k4597 · Cited by 754 · see citing works
Self-citationDeveloping clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
Self-citationMinimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400 · see citing works
Self-citationMinimum sample size for developing a multivariable prediction model: Part I – Continuous outcomes
Riley, R. D., Snell, K. I. E., Ensor, J., Burke, D. L., Harrell, F. E., Moons, K. G. M., & Collins, P. G. S. (2018) · Statistics in Medicine · doi:10.1002/sim.7993 · Cited by 372 · see citing works
Self-citationMulder, F. I., Candeloro, M., Kamphuisen, P. W., Nisio, M. D., Bossuyt, P. M., Guman, N. A. M., Smit, K., Buller, H. R., & van Es, N. (2019) · Haematologica · doi:10.3324/haematol.2018.209114 · Cited by 332 · see citing works
Guo, C., Ashrafian, H., Ghafur, S., Fontana, G., Gardner, C., & Prime, M. S. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00314-2 · Cited by 322 · see citing works
Development of type 2 diabetes mellitus in people with intermediate hyperglycaemia
Richter, B., Hemmingsen, B., Metzendorf, M., & Takwoingi, Y. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012661.pub2 · Cited by 285 · see citing works
A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes
Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., Collins, G. S., & Moons, K. G. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218785504 · Cited by 278 · see citing works
“For this reason, it is often helpful to investigate potential sources of heterogeneity in the predictive performance by performing a meta-regression or subgroup analysis.(14,61) Common sources of heterogeneity are differences in study characteristics,(62) differences in study quality, or differences in baseline risk a…”
BackgroundMethodInfluentialSelf-citationDamen, J. A., Pajouheshnia, R., Heus, P., Moons, K. G. M., Reitsma, J. B., Scholten, R. J. P. M., Hooft, L., & Debray, T. P. A. (2019) · BMC Medicine · doi:10.1186/s12916-019-1340-7 · Cited by 250 · see citing works
Self-citationFundamentals of Clinical Data Science
Kubben, P., Dumontier, M., & Dekker, A. L. A. J. (2018) · Springer eBooks · doi:10.1007/978-3-319-99713-1 · Cited by 240 · see citing works
Snell, K. I. E., Levis, B., Damen, J. A., Dhiman, P., Debray, T. P. A., Hooft, L., Reitsma, J. B., Moons, K. G. M., Collins, P. G. S., & Riley, R. D. (2023) · BMJ · doi:10.1136/bmj-2022-073538 · Cited by 209 · see citing works
Self-citationPinna, T., & Edwards, D. J. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.01792 · Cited by 199 · see citing works
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198 · see citing works
Self-citationBellou, V., Belbasis, L., Konstantinidis, A. K., Tzoulaki, I., & Evangelou, E. (2019) · BMJ · doi:10.1136/bmj.l5358 · Cited by 197 · see citing works
Guidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02255-9 · Cited by 190 · see citing works
Mahmoudi, E., Kamdar, N. S., Kim, N., Gonzales, G., Singh, K., & Waljee, A. K. (2020) · BMJ · doi:10.1136/bmj.m958 · Cited by 187 · see citing works
Machine learning for genetic prediction of psychiatric disorders: a systematic review
Bracher‐Smith, M., Crawford, K. L., & Escott‐Price, V. (2020) · Molecular Psychiatry · doi:10.1038/s41380-020-0825-2 · Cited by 159 · see citing works
Dretzke, J., Chuchu, N., Agarwal, R., Herd, C. P., Chua, W., Fabritz, L., Bayliss, S. E., Kotecha, D., Deeks, J. J., Kirchhof, P. F., et al. (2020) · EP Europace · doi:10.1093/europace/euaa041 · Cited by 146 · see citing works
Sufriyana, H., Husnayain, A., Chen, Y.-L., Kuo, C.-Y., Singh, O., Yeh, T.-Y., Wu, Y., & Su, E. C. (2020) · JMIR Medical Informatics · doi:10.2196/16503 · Cited by 145 · see citing works
Huang, Y., Talwar, A., Chatterjee, S., & Aparasu, R. R. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01284-z · Cited by 139 · see citing works
Jenkins, D. A., Martin, G. P., Sperrin, M., Riley, R. D., Debray, T. P. A., Collins, G. S., & Peek, N. (2021) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00090-3 · Cited by 131 · see citing works
Self-citationSnell, K. I. E., Ensor, J., Debray, T. P. A., Moons, K. G. M., & Riley, R. D. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280217705678 · Cited by 130 · see citing works
Self-citationZou, Y., Zou, X., Zheng, S., Tang, H., Zhang, L., Liu, P., & Xie, X. (2020) · Therapeutic Advances in Medical Oncology · doi:10.1177/1758835920940928 · Cited by 129 · see citing works
Snell, K. I. E., Archer, L., Ensor, J., Bonnett, L. J., Debray, T. P. A., Phillips, B., Collins, P. G. S., & Riley, R. D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.011 · Cited by 125 · see citing works
Self-citationHarnessing big data to characterize immune-related adverse events
Jing, Y., Yang, J., Johnson, D. B., Moslehi, J. J., & Han, L. (2022) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-021-00597-8 · Cited by 125 · see citing works
van Es, N., Ventresca, M., Nisio, M. D., Zhou, Q. X., Noble, S., Crowther, M. A., Briel, M., Garcia, D. A., Lyman, G. H., Macbeth, F. R., et al. (2020) · Journal of Thrombosis and Haemostasis · doi:10.1111/jth.14824 · Cited by 122 · see citing works
How to conduct a systematic review and meta-analysis of prognostic model studies
Damen, J. A., Moons, K. G. M., van Smeden, M., & Hooft, L. (2022) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2022.07.019 · Cited by 113 · see citing works
Self-citationBiomarkers of Vascular Inflammation for Cardiovascular Risk Prognostication
Antonopoulos, A. S., Angelopoulos, A., Papanikolaou, P., Simantiris, S., Oikonomou, E. K., Vamvakaris, K., Koumpoura, A., Farmaki, M., Trivella, M., Vlachopoulos, C. V., et al. (2021) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2021.09.014 · Cited by 110 · see citing works
Dhiman, P., Ma, J., Navarro, C. L. A., Speich, B., Bullock, G. S., Damen, J. A., Kirtley, S., Hooft, L., Riley, R. D., CALSTER, B. V., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.024 · Cited by 109 · see citing works
Self-citationHuang, Y., Li, J., Li, M., & Aparasu, R. R. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-02078-1 · Cited by 108 · see citing works
Association Between Sex and Immune-Related Adverse Events During Immune Checkpoint Inhibitor Therapy
Jing, Y., Zhang, Y., Wang, J., Li, K., Chen, X., Heng, J., Gao, Q., Ye, Y., Zhang, Z., Liu, Y., et al. (2021) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djab035 · Cited by 102 · see citing works
Himmelreich, J. C. L., Veelers, L., Lucassen, W. A. M., Schnabel, R. B., Rienstra, M., van Weert, H. C. P. M., & Harskamp, R. E. (2020) · EP Europace · doi:10.1093/europace/euaa005 · Cited by 100 · see citing works
Hodgson, L. E., Sarnowski, A., Roderick, P. J., Dimitrov, B. D., Venn, R. M., & Forni, L. G. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-016591 · Cited by 100 · see citing works
Igwe, E. O., Nealon, J. R., O’Shaughnessy, P., Bowden, A., Chang, H., Ho, M., Montayre, J., Montgomery, A., Rolls, K., Chou, K., et al. (2023) · Worldviews on Evidence-Based Nursing · doi:10.1111/wvn.12649 · Cited by 98 · see citing works
Meertens, L. J. E., van Montfort, P., Scheepers, H. C. J., van Kuijk, S. M. J., Aardenburg, R., Langenveld, J., van Dooren, I. M. A., Zwaan, I. M., Spaanderman, M. E. A., & Smits, L. J. M. (2018) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1111/aogs.13358 · Cited by 91 · see citing works
Prognostic prediction models for chronic postsurgical pain in adults: a systematic review
Papadomanolakis‐Pakis, N., Uhrbrand, P., Haroutounian, S., & Nikolajsen, L. (2021) · Pain · doi:10.1097/j.pain.0000000000002261 · Cited by 89 · see citing works
Takada, T., Nijman, S. W. J., Denaxas, S., Snell, K. I. E., Uijl, A., Nguyen, T., Asselbergs, F. W., & Debray, T. P. A. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.03.025 · Cited by 89 · see citing works
Self-citationVernooij, L. M., van Klei, W. A., Moons, K. G. M., Takada, T., van Waes, J. A. R., & Damen, J. A. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013139.pub2 · Cited by 89 · see citing works
Self-citationLiu, W., Laranjo, L., Klimis, H., Chiang, J., Yue, J., Marschner, S. L., Quiroz, J. C., Jorm, L. R., & Chow, C. K. (2023) · European Heart Journal - Quality of Care and Clinical Outcomes · doi:10.1093/ehjqcco/qcad017 · Cited by 84 · see citing works
Systematic review of prediction models for pulmonary tuberculosis treatment outcomes in adults
Peetluk, L. S., Ridolfi, F. M., Rebeiro, P. F., Liu, D., Rolla, V. C., & Sterling, T. R. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044687 · Cited by 83 · see citing works
Nissen, S. K., Fournaise, A., Lauridsen, J. T., Ryg, J., Nickel, C. H., Gudex, C. M., Brabrand, M., & Andersen‐Ranberg, K. (2020) · BMC Geriatrics · doi:10.1186/s12877-020-01850-y · Cited by 81 · see citing works
Ogonah, M. G. T., Seyedsalehi, A., Whiting, D., & Fazel, S. (2023) · The Lancet Psychiatry · doi:10.1016/s2215-0366(23)00256-0 · Cited by 81 · see citing works
Gerry, S., Birks, J., Bonnici, T., Watkinson, P., Kirtley, S., & Collins, P. G. S. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-019268 · Cited by 80 · see citing works
Chowdhury, M. Z. I., Naeem, I., Quan, H., Leung, A. A. C., Sikdar, K. C., O’Beirne, M., & Turin, T. C. (2022) · PLoS ONE · doi:10.1371/journal.pone.0266334 · Cited by 78 · see citing works
Miles, J., Turner, J., Jacques, R., Williams, J., & Mason, S. M. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00084-1 · Cited by 77 · see citing works
Predicting relapse or recurrence of depression: systematic review of prognostic models
Moriarty, A. S., Meader, N., Snell, K. I. E., Riley, R. D., Paton, L. W., Dawson, S., Hendon, J., Chew-Graham, C. A., Gilbody, S., Churchill, R., et al. (2022) · The British Journal of Psychiatry · doi:10.1192/bjp.2021.218 · Cited by 71 · see citing works
Self-citationMachine learning techniques for mortality prediction in emergency departments: a systematic review
Naemi, A., Schmidt, T., Mansourvar, M., Naghavi‐Behzad, M., Ebrahimi, A., & Wiil, U. K. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-052663 · Cited by 67 · see citing works
Characterization of Risk Prediction Models for Acute Kidney Injury
Feng, Y., Wang, Y., Jun, M., Pu, L., Weisbord, S. D., Bellomo, R., Hong, D., & Gallagher, M. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.13359 · Cited by 65 · see citing works
Predicting falls in community-dwelling older adults: a systematic review of prognostic models
Gade, G. V., Jørgensen, M. G., Ryg, J., Riis, J., Thomsen, K., Masud, T., & Andersen, S. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044170 · Cited by 64 · see citing works
de Jong, V. M. T., Rousset, R. Z., Antonio‐Villa, N. E., Buenen, A. G., CALSTER, B. V., Bello‐Chavolla, O. Y., Brunskill, N. J., Ćurčin, V., Damen, J. A., Fermín‐Martínez, C. A., et al. (2022) · BMJ · doi:10.1136/bmj-2021-069881 · Cited by 64 · see citing works
Self-citationSlieker, R. C., van der Heijden, A. A. W. A., Siddiqui, M. K., Langendoen‐Gort, M., Nijpels, G., Herings, R. M. C., Feenstra, T., Moons, K. G. M., Bell, S., Elders, P. J. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2134 · Cited by 63 · see citing works
Self-citationPrognostic models for mortality risk in patients requiring ECMO
Pladet, L. C. A., Barten, J. M. M., Vernooij, L. M., Kraemer, C. V. E., Bunge, J. J. H., Scholten, E., Montenij, L. J., Kuijpers, M., Donker, D. W., Cremer, O. L., et al. (2023) · Intensive Care Medicine · doi:10.1007/s00134-022-06947-z · Cited by 63 · see citing works
Prediction models for depression risk among older adults: systematic review and critical appraisal
Tan, J., Ma, C., Zhu, C., Wang, Y., Zou, X., Li, H., Li, J., He, Y., & Wu, C. (2022) · Ageing Research Reviews · doi:10.1016/j.arr.2022.101803 · Cited by 62 · see citing works
Debray, T. P. A., Collins, P. G. S., Riley, R. D., Snell, K. I. E., CALSTER, B. V., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071058 · Cited by 62 · see citing works
Self-citationKareemi, H., Vaillancourt, C., Rosenberg, H., Fournier, K., & Yadav, K. (2020) · Academic Emergency Medicine · doi:10.1111/acem.14190 · Cited by 61 · see citing works
Risk Prediction Models for Cardiotoxicity of Chemotherapy Among Patients With Breast Cancer
Kabore, E., MacDonald, C., Kaboré, A., Didier, R. J., Arveux, P., Méda, N., Boutron‐Ruault, M., & Guénancia, C. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.0569 · Cited by 60 · see citing works
Shejule, P., & Pekkat, S. (2022) · Journal of Water and Climate Change · doi:10.2166/wcc.2022.302 · Cited by 59 · see citing works
Rueegg, M., Nissen, S. K., Brabrand, M., Kaeppeli, T., Dreher, T., Carpenter, C. R., Bingisser, R. M., & Nickel, C. H. (2022) · Academic Emergency Medicine · doi:10.1111/acem.14460 · Cited by 58 · see citing works
Kolk, M. Z. H., Deb, B., Ruipérez-Campillo, S., Bhatia, N. K., Clopton, P., Wilde, A. A. M., Narayan, S. M., Knops, R. E., & Tjong, F. V. Y. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104462 · Cited by 57 · see citing works
Orcutt, X., Chen, K., Mamtani, R., Long, Q., & Parikh, R. B. (2025) · Nature Medicine · doi:10.1038/s41591-024-03352-5 · Cited by 57 · see citing works
Developing more generalizable prediction models from pooled studies and large clustered data sets
de Jong, V. M. T., Moons, K. G. M., Eijkemans, M. J. C., Riley, R. D., & Debray, T. P. A. (2021) · Statistics in Medicine · doi:10.1002/sim.8981 · Cited by 56 · see citing works
Self-citationvan der Endt, V. H. W., Milders, J., de Vries, B. B. L. P., Trines, S. A. I. P., Groenwold, R. H. H., Dekkers, O. M., Trevisan, M., Carrero, J. J., van Diepen, M., Dekker, F. W., et al. (2022) · EP Europace · doi:10.1093/europace/euac096 · Cited by 54 · see citing works
Khalifa, M., Magrabi, F., & Gallego, B. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0940-7 · Cited by 54 · see citing works
Guidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · JBI Evidence Synthesis · doi:10.11124/jbies-23-00139 · Cited by 53 · see citing works
Risk prediction models for maternal mortality: A systematic review and meta-analysis
Aoyama, K., D'Souza, R. D., Pinto, R. L., Ray, J. G., Hill, A. D., Scales, D. C., Lapinsky, S. E., Seaward, G. R., Hladunewich, M., Shah, P. S., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0208563 · Cited by 51 · see citing works
Aldin, A., Umlauff, L., Estcourt, L. J., Collins, P. G. S., Moons, K. G. M., Engert, A., Kobe, C., von Tresckow, B., Haque, M., Foroutan, F., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012643.pub3 · Cited by 51 · see citing works
Self-citationPrediction models for clinical outcome after cochlear implantation: a systematic review
Velde, H. M., Rademaker, M. M., Damen, J. A., Smit, A. L., & Stegeman, I. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.04.005 · Cited by 50 · see citing works
Self-citationFrondelius, T., Atkova, I., Miettunen, J., Rello, J., Vesty, G. M., Chew, H. S. J., & Jansson, M. M. (2023) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2023.11.009 · Cited by 50 · see citing works
Su, N., Lagerweij, M. D., & van der Heijden, G. J. M. G. (2021) · Journal of Dentistry · doi:10.1016/j.jdent.2021.103664 · Cited by 49 · see citing works
Fu, H., Hou, D., Xu, R., You, Q., Li, H., Yang, Q., Wang, H., Gao, J., & Bai, D. (2023) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2023.104623 · Cited by 49 · see citing works
van Doorn, S., Debray, T. P. A., Kaasenbrood, F., Hoes, A. W., Rutten, F. H., Moons, K. G. M., & Geersing, G. (2017) · Journal of Thrombosis and Haemostasis · doi:10.1111/jth.13690 · Cited by 48 · see citing works
Self-citationKreuzberger, N., Damen, J. A., Trivella, M., Estcourt, L. J., Aldin, A., Umlauff, L., Vazquez-Montes, M., Wolff, R. F., Moons, K. G. M., Monsef, I., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012022.pub2 · Cited by 48 · see citing works
Self-citationRisk models for recurrence and survival after kidney cancer: a systematic review
Usher‐Smith, J. A., Li, L., Roberts, L. E., Harrison, H., Rossi, S. H., Sharp, S. J., Coupland, C. A. C., Hippisley‐Cox, J., Griffin, S. J., Klatte, T., et al. (2021) · British Journal of Urology · doi:10.1111/bju.15673 · Cited by 48 · see citing works
Externally validated prediction models for pre‐eclampsia: systematic review and meta‐analysis
Tiruneh, S. A., Vu, T. T. T., Moran, L., Callander, E. J., Allotey, J., Thangaratinam, S., Rolnik, D. L., Teede, H. J., Wang, R., & Enticott, J. (2023) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.27490 · Cited by 47 · see citing works
Lindroth, H. L., Bratzke, L. C., Twadell, S., Rowley, P. A., Kildow, J., Danner, M., Turner, L., Hernandez, B. A., Brown, R., & Sanders, R. D. (2019) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.5104 · Cited by 47 · see citing works
Medina‐Lara, A., Grigore, B., Lewis, R. A., Peters, J., Price, S., Landa, P., Robinson, S., Neal, R. D., Hamilton, W. T., & Spencer, A. E. (2020) · Health Technology Assessment · doi:10.3310/hta24660 · Cited by 47 · see citing works
Kretzer, S., Lawrence, A. J., Pollard, R., Ma, X., Chen, P. J., Amasi-Hartoonian, N., Pariante, C. M., Vallée, C., Meaney, M., & Dazzan, P. (2024) · Biological Psychiatry · doi:10.1016/j.biopsych.2024.06.012 · Cited by 46 · see citing works
Wang, J., Zhang, Z., Li, Y., Xu, Y., Wan, K., & Chen, Y. (2019) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2019.05.004 · Cited by 46 · see citing works
Barca, L. V., Navas, E., Fernández‐Hidalgo, N., Mur, J. L. M., Muriel, A., Fernandez-Felix, B. M., Hycka, J. M., Rodríguez-Roda, J., & Menéndez, J. L. (2019) · Infection · doi:10.1007/s15010-019-01338-x · Cited by 46 · see citing works
Damen, J. A., Debray, T. P. A., Pajouheshnia, R., Reitsma, J. B., Scholten, R. J. P. M., Moons, K. G. M., & Hooft, L. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-026160 · Cited by 45 · see citing works
Self-citationGates, M., Pillay, J., Nuspl, M., Wingert, A., Vandermeer, B., & Hartling, L. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02181-w · Cited by 46 · see citing works
Zhong, X., Long, H., Su, L., Zheng, R., Wang, W., Duan, Y., Hu, H., Lin, M., & Xie, X. (2022) · Abdominal Radiology · doi:10.1007/s00261-022-03496-3 · Cited by 44 · see citing works
Prediction Models for Intravenous Immunoglobulin Resistance in Kawasaki Disease: A Meta-analysis
Kuniyoshi, Y., Tsujimoto, Y., Banno, M., Taito, S., Ariie, T., Takahashi, N., Tokutake, H., & Takada, T. (2023) · PEDIATRICS · doi:10.1542/peds.2022-059175 · Cited by 44 · see citing works
Delivering preexposure prophylaxis to pregnant and breastfeeding women in Sub-Saharan Africa
Davey, D. J., Bekker, L., Gorbach, P. M., Coates, T. J., & Myer, L. (2017) · AIDS · doi:10.1097/qad.0000000000001604 · Cited by 44 · see citing works
Forecasting future global food demand: A systematic review and meta-analysis of model complexity
Flies, E. J., Brook, B. W., Blomqvist, L., & Buettel, J. C. (2018) · Environment International · doi:10.1016/j.envint.2018.07.019 · Cited by 44 · see citing works
Tiruneh, S. A., Vu, T. T. T., Rolnik, D. L., Teede, H. J., & Enticott, J. (2024) · Current Hypertension Reports · doi:10.1007/s11906-024-01297-1 · Cited by 43 · see citing works
Barreñada, L., Ledger, A., Dhiman, P., Collins, G., Wynants, L., Verbakel, J. Y., Timmerman, D., Valentin, L., & Van Calster, B. (2024) · BMJ Medicine · doi:10.1136/bmjmed-2023-000817 · Cited by 42 · see citing works
Bronchopulmonary dysplasia prediction models: a systematic review and meta-analysis with validation
Kwok, T. C., Batey, N., Luu, K. L., Prayle, A., & Sharkey, D. (2023) · Pediatric Research · doi:10.1038/s41390-022-02451-8 · Cited by 42 · see citing works
Pan, Z., Zhang, R., Shen, S., Lin, Y., Zhang, L., Wang, X., Ye, Q., Wang, X., Chen, J., Zhao, Y., et al. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104443 · Cited by 42 · see citing works
When and how to use data from randomised trials to develop or validate prognostic models
Pajouheshnia, R., Groenwold, R. H. H., Peelen, L. M., Reitsma, J. B., & Moons, K. G. M. (2019) · BMJ · doi:10.1136/bmj.l2154 · Cited by 42 · see citing works
Self-citationYao, X., Flórez, I. D., Zhang, P., Zhang, C., Zhang, Y., Wang, C., Liu, X., Nie, X., Wei, B., & Ghert, M. A. (2020) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12384 · Cited by 42 · see citing works
Allotey, J., Laivuori, H. M., Snell, K. I. E., Smuk, M., Hooper, R., Chan, C. L., Ahmed, A., Chappell, L. C., von Dadelszen, P., Dodds, J. P., et al. (2020) · Health Technology Assessment · doi:10.3310/hta24720 · Cited by 41 · see citing works
Self-citationIntraoperative risk factors of acute kidney injury after liver transplantation
Berkowitz, R. J., Engoren, M., Mentz, G. B., Sharma, P., Kumar, S. S., Davis, R. P., Kheterpal, S., Sonnenday, C. J., & Douville, N. J. (2022) · Liver Transplantation · doi:10.1002/lt.26417 · Cited by 40 · see citing works
Protease activity as a prognostic factor for wound healing in venous leg ulcers
Westby, M. J., Dumville, J. C., Stubbs, N. C., Norman, G., Wong, J. K. F., Cullum, N. A., & Riley, R. D. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012841.pub2 · Cited by 40 · see citing works
Self-citationPrediction models for postoperative pulmonary complications: a systematic review and meta-analysis
Huang, Z., Han, Y., Zhuang, H., Jiang, J., Zhou, C., & Yu, H. (2025) · British Journal of Anaesthesia · doi:10.1016/j.bja.2025.04.025 · Cited by 38 · see citing works
- Authors
- Debray, T. P. A., Damen, J. A. A. G., Snell, K. I. E., Ensor, J., Hooft, L., Reitsma, J. B., Riley, R. D., & Moons, K. G. M.
- Affiliations
- University Medical Center Utrecht (NL)
- Keele University (GB)
- Corresponding author
- Debray, Thomas P A
- ORCID
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2017-01-05
- Volume
- 356
- Pages
- i6460
- ISSN
- 1756-1833 (ISSN-L)
- OpenAlex
- W2570878489
- Cited by
- 657
- References
- 48
- Citations by year
- 2026: 109 · 2025: 98 · 2024: 94 · 2023: 81 · 2022: 64 · 2021: 82 · 2020: 56 · 2019: 33 · 2018: 27 · 2017: 11 · 2012: 1
- FWCI
- 8.62
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- BMJ (publishedVersion)
- PubMed (publishedVersion)
- Pure Amsterdam UMC (publishedVersion)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- HighWire Press Open Archive (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Funders
- Medical Research Council (MR/L004933/1, G0902393)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- prediction model validation, systematic reviews, meta-analysis, predictive performance, discrimination (statistics), calibration, EuroSCORE, postoperative mortality, coronary artery bypass grafting, risk prediction model
- MeSH terms
- Systematic Reviews as Topic, Coronary Artery Bypass, Decision Support Techniques, Humans, Meta-Analysis as Topic, Risk Assessment, Validation Studies as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
