PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Clinical prediction models combine multiple predictors to estimate risk for the presence of a particular condition (diagnostic models) or the occurrence of a certain event in the future (prognostic models). PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool for assessing the risk of bias (ROB) and applicability of diagnostic and prognostic prediction model studies, was developed by a steering group that considered existing ROB tools and reporting guidelines. The tool was informed by a Delphi procedure involving 38 experts and was refined through piloting. PROBAST is organized into the following 4 domains: participants, predictors, outcome, and analysis. These domains contain a total of 20 signaling questions to facilitate structured judgment of ROB, which was defined to occur when shortcomings in study design, conduct, or analysis lead to systematically distorted estimates of model predictive performance. PROBAST enables a focused and transparent approach to assessing the ROB and applicability of studies that develop, validate, or update prediction models for individualized predictions. Although PROBAST was designed for systematic reviews, it can be used more generally in critical appraisal of prediction model studies. Potential users include organizations supporting decision making, researchers and clinicians who are interested in evidence-based medicine or involved in guideline development, journal editors, and manuscript reviewers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329 · cited 3×
“A domain-based structure was adopted similar to that used in other risk of bias tools such as the revised Cochrane Risk of Bias tool,(29) QUADAS-2,(36) ROBINSI(30) and ROBIS.”
BackgroundMethodSelf-citationQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199 · cited 3×
“A domain-based structure was adopted similar to that used in other risk of bias tools such as the revised Cochrane Risk of Bias tool,(29) QUADAS-2,(36) ROBINSI(30) and ROBIS.”
InfluentialSelf-citationCochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,178
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · cited 11×
“A multivariable prediction model is defined as any combination or equation of two or more predictors for estimating the probability or risk for an individual.(7, 8, 33-35)”
InfluentialSelf-citation2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.005 · Cited by 4,945
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Circulation · doi:10.1161/01.cir.0000437741.48606.98 · Cited by 3,697
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Self-citationAssessing Bias in Studies of Prognostic Factors
Hayden, J. A., van der Windt, D. A., Cartwright, J. L., Côté, P., & Bombardier, C. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-4-201302190-00009 · Cited by 3,287 · cited 1×
“(25, 26) The QUIPS tool has been developed to assess the risk of bias in predictor finding (prognostic factor) studies.(28) The methodological quality of studies investigating the impact of a prediction model using a comparative randomised design can be assessed using the revised Cochrane risk of bias tool (ROB 2.”
MethodCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Self-citationSystematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
Cited by
Items 601–700 of 1,010
Davenport, C. F., Rai, N., Sharma, P., Deeks, J. J., Berhane, S., Mallett, S., Saha, P., Solanki, R., Bayliss, S. E., Snell, K. I. E., et al. (2022) · Cancers · doi:10.3390/cancers14153621 · Cited by 26
“Tailoring included a question about follow up of index test negatives (>12 months), two questions relevant to multivariable model/composite index test validation studies from a quality checklist for prognostic studies [13], and whether borderline ovarian tumours (BOT) were appropriately handled, i.”
BackgroundSelf-citationKlunder, J. H., Panneman, S. L., Wallace, E., de Vries, R., Joling, K. J., Maarsingh, O. R., & van Hout, H. P. J. (2022) · PLoS ONE · doi:10.1371/journal.pone.0275116 · Cited by 23
“The Prediction model Risk of Bias ASsessment Tool (PROBAST; S2 File) was used to assess risk of bias and applicability, of which the latter addresses whether the primary study matches the review question [25].”
The ABC recommendations for validation of supervised machine learning results in biomedical sciences
Chicco, D., & Jurman, G. (2022) · Frontiers in Big Data · doi:10.3389/fdata.2022.979465 · Cited by 22
“In the past, several studies proposed rules and guidelines to develop more effective and efficient predictive models in medical informatics and computational epidemiology (Steyerberg and Vergouwe, 2014, Riley et al., 2016, 2021; Bonnett et al., 2019; Wolff et al., 2019; Navarro et al., 2021; Van Calster et al., 2021).”
BackgroundLi, H., Tian, Y., Niu, H., He, L., Cao, G., Zhang, C., Kaiweisierkezi, K., & Luo, Q. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.988287 · Cited by 22
“Assessment of risk of bias and applicability Risk of bias (ROB) and applicability was assessed using the Prediction Study Risk Of Bias Assessment Tool (PROBAST) (30).”
Osimo, E. F., Perry, B. I., Mallikarjun, P. K., Pritchard, M., Lewis, J., Katunda, A., Murray, G. K., Pérez, J. M., Jones, P. B., Cardinal, R. N., et al. (2023) · Nature Mental Health · doi:10.1038/s44220-022-00001-z · Cited by 24
The Role of AI in Breast Cancer Lymph Node Classification: A Comprehensive Review
Vrdoljak, J., Krešo, A., Kumrić, M., Martinović, D., Cvitković, I., Grahovac, M., Vickov, J., Bukić, J., & Božić, J. (2023) · Cancers · doi:10.3390/cancers15082400 · Cited by 22
Lee, M.-H., Shomanov, A., Kudaibergenova, M., & Viderman, D. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12103446 · Cited by 22
“The PROBAST assessment tool [103] also revealed that all included studies had a high risk of bias, thereby indicating that the model’s performance in practice may be lower than reported.”
MethodShakibfar, S., Nyberg, F., Li, H., Zhao, J., Nordeng, H., Sandve, G. K., Pavlović, M., Hajiebrahimi, M., Andersen, M. T., & Sessa, M. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1183725 · Cited by 28
Guitton, T., Allaume, P., Rabilloud, N., Rioux-Leclercq, N. C., Henno, S., Turlin, B., Galibert, M., Lièvre, A., Lespagnol, A., Pécot, T., et al. (2023) · Diagnostics · doi:10.3390/diagnostics14010099 · Cited by 23
“We also evaluated each study with PROBAST [41].”
MethodMcCoy, R. G., SWARNA, K. S., Deng, Y., Herrin, J. S., Ross, J. S., Kent, D. M., Borah, B. J., Crown, W. H., Montori, V. M., Umpierrez, G. E., et al. (2024) · JACC Advances · doi:10.1016/j.jacadv.2024.100852 · Cited by 24
Boko, M. F., Khanna, A. K., D’Aragon, F., Spence, J. D., Conen, D., Patel, A., Ayad, S. S., Wijeysundera, D. N., Choinière, M., Sessler, D. I., et al. (2024) · Anesthesiology · doi:10.1097/aln.0000000000005030 · Cited by 27
Chen, J., Chen, A., Yang, S., Liu, J., Xie, C., & Jiang, H. (2024) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2024.110325 · Cited by 27
Bravo, L., Simoes, J. F. F., Cardoso, V. R., Adisa, A., Aguilera, M. L., Arnaud, A. P., Biccard, B. M., Calvache, J. A., Chernbumroong, S., Elhadi, M., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00065-7 · Cited by 30
Graham, N. J., Souter, L., & Salami, S. S. (2024) · Urologic Oncology Seminars and Original Investigations · doi:10.1016/j.urolonc.2024.06.002 · Cited by 22
Ara, J. (2025) · doi:10.63125/h7rbyt25 · Cited by 22
Veirman, E., Ryckeghem, D. M. L. V., Paepe, A. L. D., Kirtley, O. J., & Crombez, G. (2019) · PAIN Reports · doi:10.1097/pr9.0000000000000775 · Cited by 22
Cooray, S. D., Wijeyaratne, L. A., Soldatos, G., Allotey, J., Boyle, J. A., & Teede, H. J. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17093048 · Cited by 27
“The risk of bias and applicability of included models was systematically and objectively evaluated using PROBAST [20], a robust and for-purpose tool.”
MethodInfluentialChappell, F. M., Crawford, F., Horne, M., Leese, G., Martin, A., Weller, D., Boulton, A. J. M., Abbott, C. A., Monteiro‐Soares, M., Veves, A., et al. (2021) · BMJ Open Diabetes Research & Care · doi:10.1136/bmjdrc-2021-002150 · Cited by 25
Self-citationBhandari, A., Marwah, R., Smith, J., Nguyễn, D. A., Bhatti, A., Lim, C. P., & Lasocki, A. (2022) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13436 · Cited by 26
“Quality assessment The quality assessment of the studies was conducted based on the PROBAST (Prediction model Risk Of Bias ASsessment Tool), which is used to assess diagnostic test risk of bias and applicability.(22) The Checklist for Artificial Intelligence in Medical Imaging (CLAIM)(23) was also administered, as thi…”
MethodA predictive nomogram for trismus after radiotherapy for head and neck cancer
Massaccesi, M., Dinapoli, N., Fuga, V., Rupe, C., Panfili, M., Calandrelli, R., Settimi, S., Olivieri, M., Bartoli, F. B., Mazzarella, C., et al. (2022) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2022.05.031 · Cited by 22
Testing Clinical Prediction Models
Wang, J., & Li, Y. (2020) · JAMA · doi:10.1001/jama.2020.19389 · Cited by 22
Towards reliable forecasting of healthcare capacity needs: A scoping review and evidence mapping
Grøntved, S., Kirkeby, M. J., Johnsen, S. P., Mainz, J., Valentin, J. B., & Jensen, C. M. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105527 · Cited by 28
Mohammadzadeh, I., Niroomand, B., Shahnazian, Z., Ghanbarnia, R., Nouri, Z., Tajerian, A., Choubineh, T., Najafi, M., Mohammadzadeh, S., Soltani, R., et al. (2024) · Clinical Neurology and Neurosurgery · doi:10.1016/j.clineuro.2024.108668 · Cited by 21
Zhang, W., Tang, Y., Liu, H., Li, Y., Wang, C. C., Chen, S. F., Huang, J., & Xiao, X. Y. (2021) · PLoS ONE · doi:10.1371/journal.pone.0257768 · Cited by 25
“The evaluation of the risk of bias and applicability of the models, performed using PROBAST, is presented in Table 4.”
MethodRisk Prediction Models for Inadvertent Intraoperative Hypothermia: A Systematic Review
Yan, L., Yao, L., Zhao, Q., Xiao, M., Li, Y., & Min, S. (2021) · Journal of PeriAnesthesia Nursing · doi:10.1016/j.jopan.2021.02.011 · Cited by 23
Langenberger, B., Thoma, A., & Vogt, V. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01751-7 · Cited by 22
“While steps one and two consider the review question and classification of the prediction model, step three assesses the risk of bias (ROB) and applicability, and finally step four closes with an overall judgement [47].”
BackgroundInfluentialImaging to predict checkpoint inhibitor outcomes in cancer. A systematic review
ter Maat, L. S., van Duin, I. A. J., Elias, S. G., van Diest, P. J., Pluim, J. P. W., Verhoeff, J. J. C., de Jong, P. A., Leiner, T., Veta, M., & Suijkerbuijk, K. P. M. (2022) · European Journal of Cancer · doi:10.1016/j.ejca.2022.07.034 · Cited by 21
Appleyard, T., Thomas, M. J., Antcliff, D., & Peat, G. (2022) · Arthritis Care & Research · doi:10.1002/acr.25035 · Cited by 25
“For all included articles, we assessed the risk of bias across 4 predetermined domains (participants, outcome, predictors, and analysis) using the Prediction study Risk of Bias Assessment Tool (20).”
MethodPrediction model for cognitive frailty in older adults: A systematic review and critical appraisal
Huang, J., Zeng, X., Hu, M., Ning, H., Wu, S., Peng, R., & Feng, H. (2023) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2023.1119194 · Cited by 21
“Studies for model development with EPVs less than 10, while for model validation with participants less than 100, are likely to be overfitting (Wolff et al., 2019).”
MethodInfluentialArtificial intelligence-aided ultrasound in renal diseases: a systematic review
Liang, X., Du, M., & Chen, Z. (2023) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-22-1428 · Cited by 23
Astley, J., Reilly, J. M., Robinson, S., Wild, J. M., Hatton, M. Q. F., & Tahir, B. (2024) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2024.110084 · Cited by 21
Ho, L. T. F., Pugh, C., Seth, S., Arakelyan, S., Lone, N., Lyall, M. J., Anand, A., Fleuriot, J., Galdi, P., & Guthrie, B. (2024) · The Lancet Healthy Longevity · doi:10.1016/s2666-7568(23)00264-7 · Cited by 22
Legouis, D., Rinaldi, A., Malpetti, D., Arnoux, G., Verissimo, T., Faivre, A., Mangili, F., Rinaldi, A., Ruinelli, L., Pugin, J., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109271 · Cited by 25
Beak, W., Park, J., & Ji, S. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e32496 · Cited by 23
van Roessel, S. R., Strijker, M., Steyerberg, E. W., Groen, J. V., Mieog, J. S. D., Groot, V. P., He, J. P., Pastena, M. D., Marchegiani, G., Bassi, C., et al. (2019) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2019.12.023 · Cited by 23
Preece, R., Dilaver, N., Waldron, C.-A., Pallmann, P., Thomas‐Jones, E., Gwilym, B. L., Norvell, D. C., Czerniecki, J. M., Twine, C. P., & Bosanquet, D. C. (2021) · European Journal of Vascular and Endovascular Surgery · doi:10.1016/j.ejvs.2021.02.038 · Cited by 21
“PROBAST assesses for bias arising from the study participants, risk predictors, outcomes and the analysis [12,13].”
Kordi, M., Dehghan, M. J., Shayesteh, A. A., & Azizi, A. (2022) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2022.100891 · Cited by 24
Predictive models for musculoskeletal injury risk: why statistical approach makes all the difference
Rhon, D. I., Teyhen, D. S., Collins, G. S., & Bullock, G. S. (2022) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2022-001388 · Cited by 20
Clinical Promise of Brain-Phenotype Modeling: A Review
Greene, A. S., & Constable, R. T. (2023) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2023.1419 · Cited by 20
Liu, X., Liu, X., Jin, C., Luo, Y., Yang, L., Ning, X., Zhuo, C., & Xiao, F. (2024) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2024.07.005 · Cited by 23
Chen, K., Zhang, Y., Zhang, L., Zhang, W., & Chen, Y. (2024) · Journal of Thrombosis and Haemostasis · doi:10.1016/j.jtha.2024.11.001 · Cited by 20
Machine Learning Approaches in High Myopia: Systematic Review and Meta-Analysis
Zuo, H., Huang, B., He, J., Li, F., & Huang, M. (2025) · Journal of Medical Internet Research · doi:10.2196/57644 · Cited by 24
“The risk of bias in the eligible studies was appraised by two independent reviewers (HZ and LF) using the prediction model risk of bias assessment tool [16].”
Angelantonio, E. D., Pennells, L. A. M., Abdelhamid, M., Aboyans, V., Asteggiano, R., Čelutkienė, J., Grobbee, D. E., Iung, B., Jüni, P., McEvoy, J. W., et al. (2025) · European Heart Journal · doi:10.1093/eurheartj/ehaf016 · Cited by 21
Prognostic rules for predicting cognitive syndromes following stroke: A systematic review
Drozdowska, B. A., McGill, K., McKay, M., Bartlam, R., Langhorne, P., & Quinn, T. J. (2021) · European Stroke Journal · doi:10.1177/2396987321997045 · Cited by 21
“We assessed risk of bias for each included study using the Prediction model Risk Of Bias Assessment Tool (PROBAST).(17) The tool comprises four domains: participants, predictors, outcome and analysis, each appraised separately and then considered in conjunction to make an overall judgment on risk of bias (internal val…”
MethodLocquet, M., Diep, A. N., Beaudart, C., Dardenne, N., Brabant, C., Bruyère, O., & Donneau, A. (2021) · Archives of Public Health · doi:10.1186/s13690-021-00630-3 · Cited by 20
“Two reviewers (AD and ML) assessed the quality and RoB of each included study using the Prediction model study Risk Of Bias Assessment Tool (PROBAST) [11], designed specifically for systematic reviews of diagnostic prediction models.”
BackgroundMethodInfluentialAllotey, J., Whittle, R., Snell, K. I. E., Smuk, M., Townsend, R., von Dadelszen, P., Heazell, A. E. P., Magee, L. A., Smith, G. C. S., Sandall, J., et al. (2021) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.23757 · Cited by 21
Self-citationFleet, H., Pilcher, D. V., Bellomo, R., & Coulson, T. G. (2021) · Perfusion · doi:10.1177/02676591211037025 · Cited by 21
Monahan, A. C., & Feldman, S. S. (2021) · JMIR Medical Informatics · doi:10.2196/30022 · Cited by 23
“AUROC: area under the receiver operating characteristics curve CHARMS: Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies PROBAST: Prediction Model Risk of Bias Assessment Tool Edited by G Eysenbach, C Lovis; submitted 28.04.21; peer-reviewed by A Vagelatos; com…”
InfluentialEthical Challenges of Artificial Intelligence in Health Care: A Narrative Review
Hui, A. T., Ahn, S. S., Lye, C. T., & Deng, J. (2021) · Ethics in Biology Engineering and Medicine An International Journal · doi:10.1615/ethicsbiologyengmed.2022041580 · Cited by 29
Riley, S. M., Zhang, Q., Tse, W.-Y., Connor, A., & Wei, Y. (2022) · Transplant International · doi:10.3389/ti.2022.10397 · Cited by 20
“Risk of Bias We assessed the risk of bias (RoB) in individual models using the Prediction Model Risk of Bias Assessment Tool (PROBAST) (13).”
Lans, A., Pierik, R., Bales, J. R., Fourman, M. S., Shin, D., Kanbier, L. N., Rifkin, J., DiGiovanni, W., Chopra, R. R., Moeinzad, R., et al. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102396 · Cited by 24
Østergaard, H. B., Hageman, S. H. J., Read, S. H., Owen, T., Pennells, L. A. M., Kaptoge, S. K., Petitjean, C., Xu, Z., Shi, F., McEvoy, J. W., et al. (2022) · European Journal of Preventive Cardiology · doi:10.1093/eurjpc/zwac232 · Cited by 25
“Our approach to model development and validation complies with PROBAST guidelines 23 and TRIPOD.”
Pandor, A., Daru, J., Hunt, B. J., Rooney, G., Hamilton, J., Clowes, M., Goodacre, S. W., Nelson‐Piercy, C., & Davis, S. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-065892 · Cited by 21
Liao, K. F., Wang, T., Coomber-Moore, J., Wong, D., Gomes, F., Finn, C. F., Sperrin, M., Yorke, J., & van der Veer, S. N. (2022) · BMC Cancer · doi:10.1186/s12885-022-10151-z · Cited by 22
“For prognostic model development studies, we applied the risk of bias assessment section of the Prediction model Risk Of Bias ASsessment Tool (PROBAST) [30].”
InfluentialQu, C., Luo, W., Zeng, Z., Lin, X., Gong, X., Wang, X., Zhang, Y., & Li, Y. (2022) · Heliyon · doi:10.1016/j.heliyon.2022.e11361 · Cited by 21
“Two review authors independently assessed the risk of bias in eligible trials using the QUADAS-2 tool (quality assessment of diagnostic accuracy studies) (Whiting et al., 2011) and PROBAST (prediction model risk of bias assessment tool) (Wolff et al., 2019).”
MethodLangenhuijsen, L. F. S., Janse, R. J., Venema, E., Kent, D. M., van Diepen, M., Dekker, F. W., Steyerberg, E. W., & de Jong, Y. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.04.012 · Cited by 21
A scoring tool to predict mortality and dependency after cerebral venous thrombosis
Lindgren, E., Krzywicka, K., de Winter, M. A., van Kammen, M. S., Heldner, M. R., Hiltunen, S., de Sousa, D. A., Mansour, M., Canhão, P., Ekizoğlu, E., et al. (2023) · European Journal of Neurology · doi:10.1111/ene.15844 · Cited by 20
Wang, H., Xia, Z., Xu, Y., Sun, J., & Wu, J. (2023) · Prostate Cancer and Prostatic Diseases · doi:10.1038/s41391-023-00704-z · Cited by 24
“SUMMARY OF EVIDENCE: The risk of bias in the included studies was evaluated using the Prediction model Risk of Bias Assessment Tool (PROBAST).”
BackgroundMethodInfluentialLuchinin, A. S. (2024) · Clinical oncohematology · doi:10.21320/2500-2139-2023-16-1-27-36 · Cited by 28
Milders, J., Ramspek, C. L., Janse, R. J., Bos, W. J. W., Rotmans, J. I., Dekker, F. W., & van Diepen, M. (2023) · Journal of the American Society of Nephrology · doi:10.1681/asn.0000000000000285 · Cited by 23
Yang, H., Lu, S., & Yang, L. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02449-9 · Cited by 21
“The methodological quality and risk of bias of each selected article were independently evaluated by the two reviewers using the Prediction model Risk-of-Bias Assessment Tool (PROBAST) [19].”
MethodSomé, N. H., Noormohammadpour, P., & Lange, S. (2024) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2024.1291362 · Cited by 20
“Based on PROBAST, two articles had a high risk of bias, twenty-seven had an unclear risk of bias (mostly due to unclear information in the predictors, outcome, and analysis domains), and the others were classi fi ed as having a low risk of bias.”
MethodInfluentialAlty, J., Goldberg, L. R., Roccati, E., Lawler, K., Bai, Q., Huang, G., Bindoff, A. D., Li, R., Wang, X., George, R. J. S., et al. (2024) · BMC Neurology · doi:10.1186/s12883-024-03609-z · Cited by 20
“The PROBAST tool [56] for assessment of prediction and diagnostic method studies confirms these development and validation strategies have a low risk of bias.”
MethodDuan, Y., Du, Y., Mu, Y., Guan, X., He, J., Zhang, J., Gu, Z., & Wang, C. (2024) · International Journal of Surgery · doi:10.1097/js9.0000000000001883 · Cited by 22
“For similar reasons, univariate analysis was not conducted to prevent overlooking important variables or mis-identifying associations due to the lack of consideration for interactions among multiple variables [15] .”
Rop, L. D., Bos, D. A. G., Stegeman, I., Holtman, G. A., Ochodo, E., Spijker, R., Otieno, J. A., Alkhlaileh, F., Deeks, J. J., Dinnes, J., et al. (2024) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015050.pub2 · Cited by 26
Burke, L., Hinkson, A., Haghnejad, V., Jones, R. L., Parker, R., & Rowe, I. A. C. (2024) · JHEP Reports · doi:10.1016/j.jhepr.2024.101227 · Cited by 29
“26 This covers four different domains: participant selection, predictors, outcome, and statistical analysis to allow for an overall risk of bias assessment.”
Morales, N. O., & García, P. (2024) · European Public & Social Innovation Review · doi:10.31637/epsir-2024-1605 · Cited by 20
Omar, M., Nassar, S., Sharif, Κ., Glicksberg, B. S., Nadkarni, G. N., & Klang, E. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2024.1512824 · Cited by 25
Prognostic models for the clinical management of malaria and its complications: a systematic review
Njim, T. N., & Tanyitiku, B. S. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-030793 · Cited by 20
Cai, R., Wu, X., Li, C., & Chao, J. (2020) · Journal of Hypertension · doi:10.1097/hjh.0000000000002442 · Cited by 20
“R.C. and X.W. conducted risk of bias assessments at the same time as data extraction using the recent version of the prediction model risk of bias assessment tool (PROBAST) [12].”
InfluentialKang, W. S., Shin, I., Pyo, J., Ahn, S., Chung, S., Ki, Y.-J., Seok, J., Park, C. Y., & Lee, S. D. (2019) · Journal of Korean Medical Science · doi:10.3346/jkms.2019.34.e318 · Cited by 20
Ribeiro, C. D. S., Uenishi, R. H., Domingues, A. D. S., Nakano, E. Y., Botelho, R. B. A., Raposo, A., & Zandonadi, R. P. (2024) · Nutrients · doi:10.3390/nu16152428 · Cited by 20
“Figures 3 and 4 present the analysis of the risk of bias and concern in the included studies, classi fi ed according to PROBAST [23].”
BackgroundMethodInfluentialvan der Heijden, T. G. W., de Ligt, K. M., Hubel, N. J., van der Mierden, S., Holzner, B., van de Poll-Franse, L. V., de Rooij, B. H., & the EORTC Quality of Life Group (2024) · Quality of Life Research · doi:10.1007/s11136-024-03820-y · Cited by 21
“The risk of bias of each study was assessed using PROBAST (Robert F. [24]).”
InfluentialAlhulaili, Z. M., Pleijhuis, R. G., Hoogwater, F. J. H., Nijkamp, M. W., & Klaase, J. M. (2025) · Langenbeck s Archives of Surgery · doi:10.1007/s00423-024-03581-9 · Cited by 19
Basanta-Torres, S., Rivas‐Fernández, M. Á., & Galdo‐Álvarez, S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.111028 · Cited by 23
Robust radiomics: a review of guidelines for radiomics in medical imaging
Avanzo, M., Soda, P., Bertolini, M., Bettinelli, A., Rancati, T., Stancanello, J., Rampado, O., Pirrone, G., & Drigo, A. (2026) · Frontiers in Radiology · doi:10.3389/fradi.2025.1701110 · Cited by 28
“Report sample homogeneity (19), balance (15), attrition, censoring, missing data (15, 49), and potential biases (18, 19, 34, 46) evaluated using tools such as PROBAST (50–52).”
Bradley, A., Meer, R. V. D., & McKay, C. J. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-027192 · Cited by 19
“Risk of bias assessment of included studies was performed according to the Prediction model Risk of Bias Assessment Tool.(17)”
MethodPrediction models for COVID-19 clinical decision making
Leeuwenberg, A., & Schuit, E. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30226-0 · Cited by 21
“The dataset used for model development (n=3841) is larger than in most currently published models, and the accompanying number of patients who died (n=313) seems appropriate according to the prediction model risk of bias assessment tool (PROBAST) 5 and guidance on sample size requirements for prediction model developm…”
MethodInfluentialHaller, M. C., Aschauer, C., Wallisch, C., Leffondré, K., van Smeden, M., Oberbauer, R. W., & Heinze, G. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.01.025 · Cited by 21
Prediction models of vaginal birth after cesarean delivery: A systematic review
Deng, B., Li, Y., Chen, J., Guo, J., Tan, J., Yang, Y., & Liu, N. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2022.104359 · Cited by 20
Kim, M., Park, S., Kim, C., & Choi, M. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2022.104411 · Cited by 19
Sidey-Gibbons, C. J., Sun, C. C., Schneider, A. K., Lu, S.-C., Lu, K. H., Wright, A. A., & Meyer, L. A. (2022) · Scientific Reports · doi:10.1038/s41598-022-22614-1 · Cited by 21
Risk Models to Predict Mortality in Burn Patients: A Systematic Review and Meta-analysis
Mrad, M. A., Qurashi, A. A. A., Mardan, Q. N. M. S., Jabr, F. A. A., Almenhali, A. A., Bamakhrama, B., Alsharif, B. H., AlEtebi, R. A. A., Zarkan, A. H., Kattan, I. A., et al. (2022) · Plastic & Reconstructive Surgery Global Open · doi:10.1097/gox.0000000000004694 · Cited by 19
Use of machine learning in pediatric surgical clinical prediction tools: A systematic review
Bianco, A., Al‐Azzawi, Z. A. M., Guadagno, E., Osmanlliu, E., Gravel, J., & Poenaru, D. (2023) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2023.01.020 · Cited by 23
“The risk of bias and applicability of the included studies were assessed by two independent reviewers (AB & ZA) using the Prediction model Risk Of Bias ASsessment Tool (PROBAST) [9].”
Kaiser, I., Pfahlberg, A. B., Mathes, S., Uter, W. J. C., Diehl, K., Steeb, T., Heppt, M. V., & Gefeller, O. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12051976 · Cited by 19
“Citation: Kaiser, I.; Pfahlberg, A.B.; Mathes, S.; Uter, W.; Diehl, K.; Steeb, T.; Heppt, M.V.; Gefeller, O. Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific R…”
MethodInfluentialLu, D., Long, X., Fu, W., Лю, Б., Zhou, X., & Sun, S. (2023) · Journal of Cancer Research and Clinical Oncology · doi:10.1007/s00432-023-04967-w · Cited by 20
“We evaluated the risk of bias for the included original studies using PROBAST, a tool developed by Moons Karel et al. (Wolff et al. 2019) at Utrecht University in the Netherlands in 2019, specifically for evaluating basic studies to develop, validate, and update prognostic or diagnostic multifactor…”
MethodRadiomics in Carotid Plaque: A Systematic Review and Radiomics Quality Score Assessment
Hou, C., Liu, X., Du, Y., Cheng, L., Liu, L., Nie, F., Zhang, W., & He, W. (2023) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2023.06.008 · Cited by 20
Khene, Z., Kammerer‐Jacquet, S., Bigot, P., Rabilloud, N., Albiges, L. K., Margulis, V., de Crevoisier, R., Acosta, O., Rioux-Leclercq, N. C., Lotan, Y., et al. (2023) · European Urology Oncology · doi:10.1016/j.euo.2023.10.018 · Cited by 21
Li, X., Bao, H., Shi, Y., Zhu, W., Peng, Z., Yan, L., Chen, J., & Shu, X. (2023) · Medicine · doi:10.1097/md.0000000000035892 · Cited by 23
“Additionally, we adhered to the Prediction model Risk of Bias Assessment Tool (PROBAST) [20] and Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) guidelines [21] which provide a framework for reporting details essential to assessing risk of…”
MethodBortesi, J. P. T., Ranisau, J., Di, S., McGillion, M. H., Rosella, L. C., Johnson, A. E. W., Devereaux, P. J., & Petch, J. (2023) · Journal of Medical Internet Research · doi:10.2196/52880 · Cited by 23
“…the included reports employed TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) guidelines using the TRIPOD adherence assessment form [39], and used the PROBAST (Prediction Model Risk of Bias Assessment Tool) to conduct critical appraisal [40].”
Clinical Decision Support System Used in Spinal Disorders: Scoping Review
Toh, Z. A., Berg, B., Han, Q., Hey, H. W. D., Pikkarainen, M. A., Grotle, M., & He, H. G. (2024) · Journal of Medical Internet Research · doi:10.2196/53951 · Cited by 21
Clinical Applications of Machine Learning
Mateussi, N., Rogers, M. P., Grimsley, E. A., Read, M. D., Parikh, R., Pietrobon, R. S., & Kuo, P. C. (2024) · Annals of Surgery Open · doi:10.1097/as9.0000000000000423 · Cited by 21
“PROBAST (Prediction model Risk Of Bias ASsessment Tool), 25 is a valuable tool for creating, validating, and improving multivariable prediction models used in diagnosis and prognosis.”
Musto, L., Smith, A. J., Pepper, C., Bujkiewicz, S., & Bown, M. J. (2024) · British journal of surgery · doi:10.1093/bjs/znae239 · Cited by 19
“Risk-of-bias assessment was conducted using the Prediction model Risk Of Bias Assessment Tool (PROBAST) 14 , which explicitly evaluates four domains: bias arising from participants and data sources; bias owing to the predictors (assessment and definition); bias due to the outcome definition; and…”
MethodInfluentialDevelopment and Validation of a Clinical Score to Predict Epilepsy After Cerebral Venous Thrombosis
Lindgren, E., Shu, L., Simaan, N. M., Krzywicka, K., de Winter, M. A., van Kammen, M. S., Molad, J., Klein, P., Hallevi, H., Barnea, R., et al. (2024) · JAMA Neurology · doi:10.1001/jamaneurol.2024.3481 · Cited by 19
Iacoboni, G., Navarro, V., Sesques, P., Rejeski, K., Bastos‐Oreiro, M., Serpenti, F., López, A. Á. M., Iraola‐Truchuelo, J., Delgado, J., Pérez, A., et al. (2024) · Journal of Hematology & Oncology · doi:10.1186/s13045-024-01608-8 · Cited by 19
“In order to validate the possible bias, we used the PRO-BAST questionnaire (supplementary appendix-1) [37].”
MethodSiddaway, A. P., Quinlivan, L. M., Kapur, N., O’Connor, R. C., & Beurs, D. P. D. (2020) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000485 · Cited by 20
“For ML as an approach to have sustained and robust value for researchers, clinicians, and policy makers, the ML algorithm (the models) and the results both need to be presented (cf. risk of bias for prediction model studies; Wolff et al., 2019).”
MethodDevelopment and validation of early warning score systems for COVID‐19 patients
Youssef, A., Kouchaki, S., Shamout, F. E., Armstrong, J., El-Bouri, R., Taylor, T. K. F., Birrenkott, D. A., Vasey, B., Soltan, A. A. S., Zhu, T., et al. (2021) · Healthcare Technology Letters · doi:10.1049/htl2.12009 · Cited by 20
Poorthuis, M. H. F., Herings, R. A. R., Dansey, K. D., Damen, J. A., Greving, J. P., Schermerhorn, M. L., & de Borst, G. J. (2021) · Stroke · doi:10.1161/strokeaha.120.032527 · Cited by 20
Huang, Z., Aarab, G., Ravesloot, M. J. L., Zhou, N., Bosschieter, P. F. N., van Selms, M. K. A., den Haan, C., de Vries, N., Lobbezoo, F., & Hilgevoord, A. A. J. (2021) · Sleep Medicine · doi:10.1016/j.sleep.2021.10.015 · Cited by 23
Hillier, B., Scandrett, K., Coombe, A., Hernandez‐Boussard, T., Steyerberg, E. W., Takwoingi, Y., Veličković, V. M., & Dinnes, J. (2025) · PLoS Medicine · doi:10.1371/journal.pmed.1004518 · Cited by 20
“Similarly, quality assessment tools highlighting important areas for consideration in primary studies have been available for DTA studies since 2003, with an adaption to prognostic accuracy published in 2022 [103] and PROBAST for prediction model studies in 2019 [33].”
Metadata
- Authors
- Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group†
- Affiliations
- Kleijnen Systematic Reviews (United Kingdom) (GB)
- Utrecht University (NL)
- Keele University (GB)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom (G.S.C.)
- Maastricht University (NL)
- Institute of Applied Health Research, National Institute for Health Research Birmingham Biomedical Research Centre, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom (S.M.)
- ORCID
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2019-01-01
- Volume
- 170
- Issue
- 1
- Pages
- 51-58
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W2907554860
- Cited by
- 2,984
- References
- 42
- Citations by year
- 2026: 921 · 2025: 561 · 2024: 416 · 2023: 343 · 2022: 303 · 2021: 250 · 2020: 140 · 2019: 45
- FWCI
- 83.39
- Type
- journal article
- Language
- en
- Open access
- Licence
- other-oa
- www.acpjournals.org/journal/aim/text-and-data-mining (tdm)
- Locations
- Annals of Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- Bristol Research (University of Bristol) (acceptedVersion, other-oa)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- FreiDok plus (Universitätsbibliothek Freiburg) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- Radboud Repository (Radboud University) (submittedVersion)
- Explore Bristol Research (submittedVersion, other-oa)
- Apollo (University of Cambridge) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Apollo
- Funders
- National Institute for Health Research Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (ZONMW 918.10.615, ZONMW 912.08.004)
- National Institute for Health Research School for Primary Care Research (390)
- National Institute for Health Research Biomedical Research Centre, Oxford
- National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) West at University Hospitals Bristol NHS Foundation Trust
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- PROBAST, risk of bias assessment, diagnostic model, prognostic model, applicability, critical appraisal, reporting guidelines, Delphi consensus, systematic reviews, personalized prediction
- MeSH terms
- Systematic Reviews as Topic, Decision Support Techniques, Delphi Technique, Diagnosis, Humans, Prognosis, Research Design, Models, Statistical, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.