PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration
Open accessThe DOI matches its Crossref record; 194 of the 201 references it cites were found in a registry or an index.
Abstract
Prediction models in health care use predictors to estimate for an individual the probability that a condition or disease is already present (diagnostic model) or will occur in the future (prognostic model). Publications on prediction models have become more common in recent years, and competing prediction models frequently exist for the same outcome or target population. Health care providers, guideline developers, and policymakers are often unsure which model to use or recommend, and in which persons or settings. Hence, systematic reviews of these studies are increasingly demanded, required, and performed. A key part of a systematic review of prediction models is examination of risk of bias and applicability to the intended population and setting. To help reviewers with this process, the authors developed PROBAST (Prediction model Risk Of Bias ASsessment Tool) for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic. PROBAST was developed through a consensus process involving a group of experts in the field. It includes 20 signaling questions across 4 domains (participants, predictors, outcome, and analysis). This explanation and elaboration document describes the rationale for including each domain and signaling question and guides researchers, reviewers, readers, and guideline developers in how to use them to assess risk of bias and applicability concerns. All concepts are illustrated with published examples across different topics. The latest version of the PROBAST checklist, accompanying documents, and filled-in examples can be downloaded from www.probast.org.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Self-citationROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Self-citationQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349
Self-citationMeasuring the Accuracy of Diagnostic Systems
Swets, J. A. (1988) · Science · doi:10.1126/science.3287615 · Cited by 10,058
Multiple imputation using chained equations: Issues and guidance for practice
White, I. R., Royston, P., & Wood, A. M. (2011) · Statistics in Medicine · doi:10.1002/sim.4067 · Cited by 9,792
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655
Research works printed but not linked: 2
- 10.1177/0962280218785504.30032705 · doi:10.1177/0962280218785504.30032705
- Wong SS, 2003, AMIA Annu Symp Proc, 728
Cited by
Items 101–200 of 1,003
Wu, L., Lou, X., Kong, N., Xu, M., & Gao, C. (2022) · European Radiology · doi:10.1007/s00330-022-09174-8 · Cited by 76
“According to the PROBAST, all of the studies were considered to have a high ROB overall.”
MethodInfluentialBellou, V., Tzoulaki, I., van Smeden, M., Moons, K. G. M., Εvangelou, Ε., & Belbasis, L. (2021) · European Respiratory Journal · doi:10.1183/13993003.02964-2020 · Cited by 76
Self-citationShelmerdine, S. C., Arthurs, O. J., Denniston, A., & Sebire, N. J. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100385 · Cited by 66
Enabling Early Obstructive Sleep Apnea Diagnosis With Machine Learning: Systematic Review
Ferreira‐Santos, D., Amorim, P., Martins, T. S., Monteiro‐Soares, M., & Rodrigues, P. P. (2022) · Journal of Medical Internet Research · doi:10.2196/39452 · Cited by 83
Bozkurt, S., Cahan, E. M., Seneviratne, M. G., Sun, R., Lossio-Ventura, J. A., Ioannidis, J. P. A., & Hernandez-Boussard, T. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa164 · Cited by 65
Venema, E., Wessler, B. S., Paulus, J. K., Salah, R., Raman, G., Leung, L. Y., Koethe, B. C., Nelson, J., Park, J. G., van Klaveren, D., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.017 · Cited by 65
Machine learning in sudden cardiac death risk prediction: a systematic review
Barker, J., Li, X., Khavandi, S., Koeckerling, D., Mavilakandy, A. K., Pepper, C., Bountziouka, V., Chen, L., Kotb, A. I., Antoun, I., et al. (2022) · EP Europace · doi:10.1093/europace/euac135 · Cited by 72
Alabi, R. O., Elmusrati, M. S., Leivo, I. V. V., Almangush, A., & Mäkitie, A. A. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105464 · Cited by 75
Slieker, R. C., van der Heijden, A. A. W. A., Siddiqui, M. K., Langendoen‐Gort, M., Nijpels, G., Herings, R. M. C., Feenstra, T., Moons, K. G. M., Bell, S., Elders, P. J. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2134 · Cited by 70
Self-citationLandolfo, C., Bourne, T. H., Froyman, W., CALSTER, B. V., Ceusters, J., Testa, A. C., Wynants, L., Sladkevicius, P., Holsbeke, C. V., Domali, E., et al. (2022) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.26080 · Cited by 72
Trier, T. J. V., Snaterse, M., Boekholdt, S. M., Reimer, W. J. M. S. O., Hageman, S. H. J., Visseren, F. L. J., Dorresteijn, J. A. N., Peters, R. J. G., & Jorstad, H. T. (2023) · European Journal of Preventive Cardiology · doi:10.1093/eurjpc/zwad318 · Cited by 77
“6,21 Some aspects of our study warrant consideration.”
BackgroundPredicting falls in community-dwelling older adults: a systematic review of prognostic models
Gade, G. V., Jørgensen, M. G., Ryg, J., Riis, J., Thomsen, K., Masud, T., & Andersen, S. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044170 · Cited by 74
Kareemi, H., Vaillancourt, C., Rosenberg, H., Fournier, K., & Yadav, K. (2020) · Academic Emergency Medicine · doi:10.1111/acem.14190 · Cited by 66
Bayesian networks in healthcare: What is preventing their adoption?
Kyrimi, E., Dube, K., Fenton, N., Fahmi, A., Neves, M. R., Marsh, W., & McLachlan, S. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102079 · Cited by 78
van Royen, F. S.-A., Moons, K. G. M., Geersing, G., & van Smeden, M. (2022) · European Respiratory Journal · doi:10.1183/13993003.00250-2022 · Cited by 68
Self-citationPrediction models for depression risk among older adults: systematic review and critical appraisal
Tan, J., Ma, C., Zhu, C., Wang, Y., Zou, X., Li, H., Li, J., He, Y., & Wu, C. (2022) · Ageing Research Reviews · doi:10.1016/j.arr.2022.101803 · Cited by 70
Vagliano, I., Chesnaye, N. C., Leopold, J. H., Jager, K. J., Abu-Hanna, A., & Schut, M. C. (2022) · Clinical Kidney Journal · doi:10.1093/ckj/sfac181 · Cited by 59
Cai, S., Li, J., Gao, J., Pan, W., & Zhang, Y. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2022.104340 · Cited by 71
Klement, W., & El Emam, K. (2023) · Journal of Medical Internet Research · doi:10.2196/48763 · Cited by 59
de Winter, M. A., van Es, N., Buller, H. R., Visseren, F. L. J., & Nijkeuter, M. (2021) · Thrombosis Research · doi:10.1016/j.thromres.2020.12.031 · Cited by 66
Nguyen, N. H., Picetti, D., Dulai, P. S., Jairath, V., Sandborn, W. J., Ohno-Machado, L., Chen, P. L., & Singh, S. (2021) · Journal of Crohn s and Colitis · doi:10.1093/ecco-jcc/jjab155 · Cited by 68
Debray, T. P. A., Collins, G. S., Riley, R. D., Snell, K. I. E., Van Calster, B., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071058 · Cited by 57
“113 114 Use of this tool involves assessment of four domains to cover key aspects of prediction model studies: participants, predictors, outcome, and analysis.”
MethodSelf-citationXiong, S., Chen, W., Jia, X., Jia, Y., & Liu, C. (2023) · BMC Pulmonary Medicine · doi:10.1186/s12890-023-02570-w · Cited by 73
“In 2019, Wolff et al. [13] developed the prediction model risk of bias assessment tool (PROBAST), which could assess the risk of bias (ROB) and the applicability of prediction model studies.”
MethodLiu, M., Yang, X., Chen, G., Ding, Y., Shi, M., Sun, L., Huang, Z., Liu, J., Liu, T., Yan, R., et al. (2022) · Frontiers in Physiology · doi:10.3389/fphys.2022.896969 · Cited by 71
Feng, Y., Wang, Y., Jun, M., Pu, L., Weisbord, S. D., Bellomo, R., Hong, D., & Gallagher, M. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.13359 · Cited by 72
Risk Prediction Models for Kidney Cancer: A Systematic Review
Harrison, H., Thompson, R. E., Lin, Z., Rossi, S. H., Stewart, G. D., Griffin, S. J., & Usher-Smith, J. A. (2021) · European Urology Focus · doi:10.1016/j.euf.2020.06.024 · Cited by 58
Chew, E. Y., Burns, S. A., Abraham, A. G., Bakhoum, M. F., Beckman, J. A., Chui, T. Y. P., Finger, R. P., Frangi, A. F., Gottesman, R. F., Grant, M. B., et al. (2024) · Nature Reviews Cardiology · doi:10.1038/s41569-024-01060-8 · Cited by 80
“Guidelines exist for transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) 158 , to assess the risk of bias and the applicability of prediction model studies (PROBAST) 159 and their respective AI-specific extensions (TRIPOD-AI and PROBAST-AI) 160 .”
Liu, Z., Khuong, J. N., Caruana, C. B., Jackson, S. M., Campbell, R. J., Ramson, D. M., Penny‐Dimri, J. C., Kluger, M., Segal, R., & Perry, L. A. (2019) · Heart Lung and Circulation · doi:10.1016/j.hlc.2019.11.021 · Cited by 58
Artificial Intelligence in Bariatric Surgery: Current Status and Future Perspectives
Bektaş, M., Reiber, B. M. M., Pereira, J. C., Burchell, G. L., & van der Peet, D. L. (2022) · Obesity Surgery · doi:10.1007/s11695-022-06146-1 · Cited by 72
Vernooij, J. E. M., Koning, N. J., Geurts, J. W., Holewijn, S., Preckel, B., Kalkman, C. J., & Vernooij, L. M. (2023) · Anaesthesia · doi:10.1111/anae.15988 · Cited by 65
“We assessed the risk of bias of studies included in this review with the prediction model risk of bias assessment tool [23].”
MethodNicoară-Farcău, O., Han, G., Rudler, M., Angrisani, D., Monescillo, A., Torres, F., Casanovas, G., Bosch, J., Lv, Y., Dunne, P. D., et al. (2023) · Hepatology · doi:10.1097/hep.0000000000000613 · Cited by 67
“[17]”
Prediction of hypotension events with physiologic vital sign signatures in the intensive care unit
Yoon, J. H., Jeanselme, V., Dubrawski, A. W., Hravnak, M., Pinsky, M. R., & Clermont, G. (2020) · Critical Care · doi:10.1186/s13054-020-03379-3 · Cited by 68
Timmerman, S., Valentin, L., Ceusters, J., Testa, A. C., Landolfo, C., Sladkevicius, P., Holsbeke, C. V., Domali, E., Fruscio, R., Epstein, E., et al. (2022) · JAMA Oncology · doi:10.1001/jamaoncol.2022.5969 · Cited by 53
Deep Learning-Based Recurrence Prediction of Atrial Fibrillation After Catheter Ablation
Zhou, X., Nakamura, K., Sahara, N., Takagi, T., Toyoda, Y., Enomoto, Y., Hara, H., Noro, M., Sugi, K., Moroi, M., et al. (2021) · Circulation Journal · doi:10.1253/circj.cj-21-0622 · Cited by 58
van der Endt, V. H. W., Milders, J., de Vries, B. B. L. P., Trines, S. A. I. P., Groenwold, R. H. H., Dekkers, O. M., Trevisan, M., Carrero, J. J., van Diepen, M., Dekker, F. W., et al. (2022) · EP Europace · doi:10.1093/europace/euac096 · Cited by 61
Machine Learning–Based Prediction Models for Delirium: A Systematic Review and Meta-Analysis
Xie, Q., Wang, X., Pei, J., Wu, Y., Guo, Q., Su, Y., Yan, H., Nan, R., Chen, H., & Dou, X. (2022) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2022.06.020 · Cited by 58
Lu, J. H., Callahan, A., Patel, B. S., Morse, K. E., Dash, D., Pfeffer, M. A., & Shah, N. H. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.27779 · Cited by 52
Application of machine learning in predicting the risk of postpartum depression: A systematic review
Zhong, M., Zhang, H., Yu, C., Jiang, J., & Duan, X. (2022) · Journal of Affective Disorders · doi:10.1016/j.jad.2022.08.070 · Cited by 59
Rada, G., Verdugo‐Paiva, F., Ávila, C., Morel-Marambio, M., Bravo-Jeria, R., Pesce, F., Madrid, E., & Izcovich, A. (2020) · Medwave · doi:10.5867/medwave.2020.03.7867 · Cited by 55
Morley, P. T., Atkins, D. L., Finn, J., Maconochie, I. K., Nolan, J. P., Rabi, Y., Singletary, E. M., Wang, T.-L., Welsford, M., Olasveengen, T. M., et al. (2020) · Circulation · doi:10.1161/cir.0000000000000891 · Cited by 62
Prognostic models in COVID-19 infection that predict severity: a systematic review
Buttia, C., Llanaj, E., Raeisi‐Dehkordi, H., Kastrati, L., Amiri, M., Meçani, R., Taneri, P. E., Gómez‐Ochoa, S. A., Raguindin, P. F., Wehrli, F., et al. (2023) · European Journal of Epidemiology · doi:10.1007/s10654-023-00973-x · Cited by 69
“Authors are advised to follow the TRIPOD guidelines at the same time as they develop their predictive models in order to reduce their risk of bias [11, 321].”
BackgroundMethodInfluentialvan Es, N., Takada, T., Kraaijpoel, N., Klok, F. A., Stals, M. A. M., Buller, H. R., Courtney, D. M., Freund, Y., Galipienzo, J., Gal, G. L., et al. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad417 · Cited by 59
“13 Finally, several diagnostic PE models include a subjective ‘Gestalt’ variable to indicate whether ‘PE is the most likely diagnosis’, 4,5,10 which is difficult to standardize as it may vary depending on physician experience.”
MethodSelf-citationPrediction models for clinical outcome after cochlear implantation: a systematic review
Velde, H. M., Rademaker, M. M., Damen, J. A., Smit, A. L., & Stegeman, I. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.04.005 · Cited by 52
Shen, Y., & Jiang, J. (2021) · Frontiers in Pediatrics · doi:10.3389/fped.2021.712276 · Cited by 51
Carrick, R. T., Park, J. G., McGinnes, H. L., Lundquist, C. M., Brown, K., Janes, W., Wessler, B. S., & Kent, D. M. (2020) · Journal of the American Heart Association · doi:10.1161/jaha.119.017625 · Cited by 50
Kreuzberger, N., Damen, J. A., Trivella, M., Estcourt, L. J., Aldin, A., Umlauff, L., Vazquez-Montes, M., Wolff, R. F., Moons, K. G. M., Monsef, I., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012022.pub2 · Cited by 66
Self-citationAcero, J. C., Schuster, A., Zacur, E., Lange, T., Stiermaier, T., Backhaus, S. J., Thiele, H., Bueno‐Orovio, A., Lamata, P., Eitel, I., et al. (2022) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2021.11.027 · Cited by 52
Wallert, J., Boberg, J., Kaldo, V., Mataix-Cols, D., Flygare, O., Crowley, J. J., Halvorsen, M., Ben Abdesslem, F., Boman, M., Andersson, E., et al. (2022) · Translational Psychiatry · doi:10.1038/s41398-022-02133-3 · Cited by 48
“Compared to a random split of data, a temporal nonrandom split is a stronger test of model generalisability since it provides a type of external validation [35, 36] through allowing for both random noise and temporal non-random bias to invalidate developed models.”
BackgroundBüttner, F., Winters, M., Delahunt, E., Elbers, R., Lura, C. B., Khan, K. M., Weir, A., & Ardern, C. L. (2020) · British Journal of Sports Medicine · doi:10.1136/bjsports-2019-100806 · Cited by 47
Su, N., Lagerweij, M. D., & van der Heijden, G. J. M. G. (2021) · Journal of Dentistry · doi:10.1016/j.jdent.2021.103664 · Cited by 60
Romijn, M., Dhiman, P., Finken, M. J. J., van Kaam, A. H., Katz, T. A., Rotteveel, J., Schuit, E., Collins, P. G. S., Onland, W., & Torchin, H. (2023) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2023.01.024 · Cited by 53
Piccininni, M., Wechsung, M., CALSTER, B. V., Rohmann, J. L., Konigorski, S., & van Smeden, M. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104666 · Cited by 68
Lu, S.-C., Xu, C., Nguyen, C., Geng, Y., Pfob, A., & Sidey-Gibbons, C. J. (2022) · JMIR Medical Informatics · doi:10.2196/33182 · Cited by 56
Real-World Implementation of Precision Psychiatry: A Systematic Review of Barriers and Facilitators
Baldwin, H., Loebel-Davidsohn, L., Oliver, D., de Pablo, G. S., Stahl, D. R., Riper, H. M., & Fusar‐Poli, P. (2022) · Brain Sciences · doi:10.3390/brainsci12070934 · Cited by 53
Risk prediction models for survival after heart transplantation: A systematic review
Aleksova, N., Alba, A. C., Molinero, V., Connolly, K. D., Orchanian‐Cheff, A., Badiwala, M., Ross, H. J., & Posada, J. D. (2019) · American Journal of Transplantation · doi:10.1111/ajt.15708 · Cited by 49
Medina‐Lara, A., Grigore, B., Lewis, R. A., Peters, J., Price, S., Landa, P., Robinson, S., Neal, R. D., Hamilton, W. T., & Spencer, A. E. (2020) · Health Technology Assessment · doi:10.3310/hta24660 · Cited by 56
de Jong, Y., Fu, E. L., van Diepen, M., Trevisan, M., Szummer, K. E., Dekker, F. W., Carrero, J. J., & Ocak, G. (2021) · European Heart Journal · doi:10.1093/eurheartj/ehab059 · Cited by 48
Wheeler, S. M., Bryant, A., Bonney, E. A., & Howell, E. A. (2021) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2021.11.023 · Cited by 45
Cai, Y., Gong, D.-X., Tang, L.-Y., Cai, Y. H., Li, H., Jing, T.-C., Gong, M., Hu, W., Zhang, Z.-W., Zhang, X., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/47645 · Cited by 68
“It is well known that insufficient sample size will lead to the risk of bias for the study in the traditional model, and the evaluation standard of sample size has been developed by the PROBAST [106].”
Kahwati, L. C., Kistler, C. E., Booth, G., Sathe, N. A., Gordon, R. D., Okah, E. D., Wines, R., & Viswanathan, M. (2025) · JAMA · doi:10.1001/jama.2024.21653 · Cited by 58
Dsouza, V. S., Leyens, L., Kurian, J. R., Brand, A. M., & Brand, H. (2025) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2025.02.008 · Cited by 66
van der Heijden, A. A. W. A., Nijpels, G., Badloe, F. M. S., Lovejoy, H. L., Peelen, L. M., Feenstra, T., Moons, K. G. M., Slieker, R. C., Herings, R. M. C., Elders, P. J. M., et al. (2020) · Diabetologia · doi:10.1007/s00125-020-05134-3 · Cited by 56
Self-citationPredicting relapse or recurrence of depression: A systematic review of prognostic models
Moriarty, A. S., Meader, N., Snell, K. I. E., Riley, R. D., Paton, L. W., Dawson, S., Hendon, J., Chew-Graham, C. A., Gilbody, S. M., Churchill, R. C., et al. (2022) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 44
Self-citationZhong, X., Long, H., Su, L., Zheng, R., Wang, W., Duan, Y., Hu, H., Lin, M., & Xie, X. (2022) · Abdominal Radiology · doi:10.1007/s00261-022-03496-3 · Cited by 47
Abou‐Ayash, S., Fonseca, M., Pieralli, S., & Reissmann, D. R. (2023) · Clinical Oral Implants Research · doi:10.1111/clr.14065 · Cited by 48
de Jong, Y., Ramspek, C. L., van der Endt, V. H. W., Rookmaaker, M. B., Blankestijn, P. J., Vernooij, R. W. M., Verhaar, M. C., Bos, W. J. W., Dekker, F. W., Ocak, G., et al. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.03.015 · Cited by 49
Barreñada, L., Ledger, A., Dhiman, P., Collins, G., Wynants, L., Verbakel, J. Y., Timmerman, D., Valentin, L., & Van Calster, B. (2024) · BMJ Medicine · doi:10.1136/bmjmed-2023-000817 · Cited by 42
Key Aspects of Prognostic Model Development and Interpretation From a Clinical Perspective
Hoesseini, A., van Leeuwen, N., Sewnaik, A., Steyerberg, E. W., de Jong, R. J. B., Lingsma, H. F., & Offerman, M. P. J. (2021) · JAMA Otolaryngology–Head & Neck Surgery · doi:10.1001/jamaoto.2021.3505 · Cited by 43
Scoping Review on the Diagnosis, Prognosis, and Treatment of Pediatric Disorders of Consciousness
Molteni, E., Canas, L. S., Briand, M., Estraneo, A., Font, C. C., Formisano, R., Валерьевна, Ф. Е., Gosseries, O., Howarth, R., Lanteri, P., et al. (2023) · Neurology · doi:10.1212/wnl.0000000000207473 · Cited by 45
Seyedsalehi, A., & Fazel, S. (2024) · BMJ Mental Health · doi:10.1136/bmjment-2024-300990 · Cited by 56
“16 This has been emphasised in numerous methodological and reporting guidelines for prognostic modelling studies, 9 17 but almost entirely overlooked in the suicide prediction literature.”
BackgroundArdila, C. M., González‐Arroyave, D., & Tobón, S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0319460 · Cited by 57
Bai, Z., Hu, S., Wang, Y., Deng, W., Gu, N., Zhao, R., Zhang, W., Ma, Y., Wang, Z., Liu, Z., et al. (2021) · Annals of Translational Medicine · doi:10.21037/atm-21-2905 · Cited by 40
Gates, M., Pillay, J., Nuspl, M., Wingert, A., Vandermeer, B., & Hartling, L. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02181-w · Cited by 50
“Outcomes rated as critical (7–9/9) were hip fracture, clinical fragility fractures, fracture-related mortality, quality of life or wellbeing, functionality and disability, serious adverse events, and prediction model calibration (KQ2 only).”
MethodInfluentialArtificial intelligence tool development: what clinicians need to know?
Chew, B.-H., & Ngiam, K. Y. (2025) · BMC Medicine · doi:10.1186/s12916-025-04076-0 · Cited by 41
Machine Learning in Diagnosing Middle Ear Disorders Using Tympanic Membrane Images: A Meta‐Analysis
Cao, Z., Chen, F., Grais, E. M., Yue, F., Cai, Y., Swanepoel, D. W., & Zhao, F. (2022) · The Laryngoscope · doi:10.1002/lary.30291 · Cited by 54
Saeidnia, H. R., Hosseini, E., Abdoli, S., & Ausloos, M. (2024) · Library Hi Tech · doi:10.1108/lht-10-2023-0514 · Cited by 71
Artificial Intelligence and Machine Learning in Cancer Pain: A Systematic Review
Salama, V., Godinich, B. M., Geng, Y., Humbert‐Vidan, L., Maule, L., Wahid, K. A., Naser, M. A., He, R., Mohamed, A., Fuller, C. D., et al. (2024) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2024.07.025 · Cited by 58
Ram, M., Afrash, M. R., Moulaei, K., Parvin, M., Esmaeeli, E., Karbasi, Z., Heydari, S., & Sabahi, A. (2024) · BMC Cancer · doi:10.1186/s12885-024-12764-y · Cited by 40
“This quality assessment tool consists of four domains (participant selection, predictors, outcome, and analysis) and includes 20 signaling questions as described in PROBAST [27].”
Schouten, J., Kalden, M. A. C. M., van Twist, E., Reiss, I. K. M., Gommers, D. A. M. P. J., van Genderen, M. E., & Taal, H. R. (2024) · Intensive Care Medicine · doi:10.1007/s00134-024-07629-8 · Cited by 59
Ardila, C. M., Yadalam, P. K., & González‐Arroyave, D. (2024) · PeerJ · doi:10.7717/peerj.18213 · Cited by 54
“The PROBAST instrument, which assesses both the risk of bias and the applicability of prediction model research for systematic reviews, was employed to evaluate bias ( Moons et al., 2019 ).”
MethodEvidence Evaluation Process and Management of Potential Conflicts of Interest
Morley, P. T., Atkins, D. L., Finn, J., Maconochie, I. K., Nolan, J. P., Rabi, Y., Singletary, E. M., Wang, T.-L., Welsford, M., Olasveengen, T. M., et al. (2020) · Resuscitation · doi:10.1016/j.resuscitation.2020.09.011 · Cited by 42
Chee, M. L., Ong, M. E. H., Siddiqui, F. J., Zhang, Z., Lim, S. L., Ho, A. F. W., & Liu, N. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18094749 · Cited by 45
Tousignant‐Laflamme, Y., Houle, C., Cook, C. E., Naye, F., LeBlanc, A., & Décary, S. (2022) · Physical Therapy · doi:10.1093/ptj/pzac023 · Cited by 47
Gosselink, M. E., van Buren, M. C., Kooiman, J., Groen, H., Ganzevoort, W., van Hamersvelt, H. W., van der Heijden, O. W. H., van de Wetering, J., & Lely, A. T. (2022) · Kidney International · doi:10.1016/j.kint.2022.06.006 · Cited by 52
Witsell, D. L., Garrett, C. G., Yarbrough, W. G., Dorrestein, S. P., Drake, A. F., & Weissler, M. C. (1995) · PubMed · Cited by 39
Bronchopulmonary dysplasia prediction models: a systematic review and meta-analysis with validation
Kwok, T. C., Batey, N., Luu, K. L., Prayle, A., & Sharkey, D. (2023) · Pediatric Research · doi:10.1038/s41390-022-02451-8 · Cited by 48
Artificial Intelligence Support for Informal Patient Caregivers: A Systematic Review
Borna, S., Maniaci, M. J., Haider, C. R., Gomez-Cabello, C. A., Pressman, S. M., Haider, S. A., Demaerschalk, B. M., Cowart, J. B., & Forte, A. J. (2024) · Bioengineering · doi:10.3390/bioengineering11050483 · Cited by 40
“While PROBAST [64] is a widely recognized tool for assessing the risk of bias in prediction model studies, its application to our selection of studies was challenging due to the variety of AI methodologies and the broad context in which they were used.”
MethodKim, H., Goo, J., Kim, Y. T., & Park, C. (2019) · Journal of Thoracic Oncology · doi:10.1016/j.jtho.2019.12.110 · Cited by 34
Examining Bias and Reporting in Oral Health Prediction Modeling Studies
Du, M., Haag, D. G., Song, Y., Lynch, J. W., & Mittinty, M. (2020) · Journal of Dental Research · doi:10.1177/0022034520903725 · Cited by 41
Mentor, K., Ratnayake, B., Akter, N., Alessandri, G., Sen, G., French, J., Manas, D., Hammond, J. S., & Pandanaboyana, S. (2020) · World Journal of Surgery · doi:10.1007/s00268-020-05741-6 · Cited by 39
Crawford, F., Chappell, F. M., Lewsey, J., Riley, R. D., Hawkins, N., Nicolson, D., Heggie, R., Smith, M. C., Horne, M., Amanna, A. E., et al. (2020) · Health Technology Assessment · doi:10.3310/hta24620 · Cited by 45
Self-citationBarboi, C., Tzavelis, A., & Muhammad, L. N. (2022) · JMIR Medical Informatics · doi:10.2196/35293 · Cited by 37
Lossio-Ventura, J. A., Song, W., Sainlaire, M., Dykes, P. C., & Hernandez‐Boussard, T. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104739 · Cited by 39
The impact of machine learning in predicting risk of violence: A systematic review
Parmigiani, G., Barchielli, B., Casale, S., Mancini, T., & Ferracuti, S. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.1015914 · Cited by 46
“(PROBAST) (23), by two reviewers independently, in duplicate, with a third reviewer to manage any disagreements.”
BackgroundMarouf, A. A., Rokne, J. G., & Alhajj, R. (2025) · Cancers · doi:10.3390/cancers17223638 · Cited by 43
“Unlike PROBAST, a tool for assessing the risk of bias and applicability of prediction models [48,49], there is no specific scoring to interpret the quality of a review by the AMSTAR 2.0 system.”
InfluentialGeeson, C., Wei, L., & Franklin, B. D. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008335 · Cited by 44
Prognostic models for predicting relapse or recurrence of major depressive disorder in adults
Moriarty, A. S., Meader, N., Snell, K. I., Riley, R. D., Paton, L. W., Chew-Graham, C. A., Gilbody, S., Churchill, R., Phillips, R. S., Ali, S., et al. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013491.pub2 · Cited by 36
Self-citationReview of guidance papers on regression modeling in statistical series of medical journals
Wallisch, C., Bach, P., Hafermann, L., Klein, N., Sauerbrei, W., Steyerberg, E. W., Heinze, G., Rauch, G., & on behalf of topic group 2 of the STRATOS initiative (2022) · PLoS ONE · doi:10.1371/journal.pone.0262918 · Cited by 50
Smith, M., Barber, P. A., Scrivener, B. J., & Stinear, C. M. (2022) · Neurorehabilitation and neural repair · doi:10.1177/15459683221085287 · Cited by 46
Yang, S., Han, Y., Yu, C., Guo, Y., Pang, Y., Sun, D., Pei, P., Yang, L., Chen, Y., Du, H., et al. (2022) · Neurology · doi:10.1212/wnl.0000000000200139 · Cited by 39
Metadata
- Authors
- Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S.
- Affiliations
- Utrecht University (NL)
- Kleijnen Systematic Reviews (United Kingdom) (GB)
- Keele University (GB)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom (G.S.C.)
- Maastricht University (NL)
- Institute of Applied Health Research, National Institute for Health Research Birmingham Biomedical Research Centre, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom (S.M.)
- ORCID
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2019-01-01
- Volume
- 170
- Issue
- 1
- Pages
- W1-W33
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W2907638671
- Cited by
- 1,669
- References
- 201
- Citations by year
- 2026: 359 · 2025: 330 · 2024: 253 · 2023: 236 · 2022: 193 · 2021: 175 · 2020: 96 · 2019: 24
- FWCI
- 56.12
- Type
- journal article
- Language
- en
- Open access
- Licence
- other-oa
- www.acpjournals.org/journal/aim/text-and-data-mining (tdm)
- Locations
- Annals of Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- Utrecht University Repository (Utrecht University) (publishedVersion, other-oa)
- University of Birmingham Research Portal (University of Birmingham) (publishedVersion)
- Bristol Research (University of Bristol) (publishedVersion, other-oa)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Explore Bristol Research (submittedVersion, other-oa)
- Research Publications (Maastricht University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Funders
- National Institute for Health Research Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (ZONMW 918.10.615, ZONMW 912.08.004)
- National Institute for Health Research School for Primary Care Research (390)
- National Institute for Health Research Biomedical Research Centre, Oxford
- National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) West at University Hospitals Bristol NHS Foundation Trust
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- PROBAST, risk of bias assessment, applicability, prediction model, diagnostic model, prognostic model, model validation
- MeSH terms
- Systematic Reviews as Topic, Decision Support Techniques, Diagnosis, Humans, Prognosis, Research Design, Models, Statistical, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.