PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Clinical prediction models combine multiple predictors to estimate risk for the presence of a particular condition (diagnostic models) or the occurrence of a certain event in the future (prognostic models). PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool for assessing the risk of bias (ROB) and applicability of diagnostic and prognostic prediction model studies, was developed by a steering group that considered existing ROB tools and reporting guidelines. The tool was informed by a Delphi procedure involving 38 experts and was refined through piloting. PROBAST is organized into the following 4 domains: participants, predictors, outcome, and analysis. These domains contain a total of 20 signaling questions to facilitate structured judgment of ROB, which was defined to occur when shortcomings in study design, conduct, or analysis lead to systematically distorted estimates of model predictive performance. PROBAST enables a focused and transparent approach to assessing the ROB and applicability of studies that develop, validate, or update prediction models for individualized predictions. Although PROBAST was designed for systematic reviews, it can be used more generally in critical appraisal of prediction model studies. Potential users include organizations supporting decision making, researchers and clinicians who are interested in evidence-based medicine or involved in guideline development, journal editors, and manuscript reviewers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329 · cited 3×
“A domain-based structure was adopted similar to that used in other risk of bias tools such as the revised Cochrane Risk of Bias tool,(29) QUADAS-2,(36) ROBINSI(30) and ROBIS.”
BackgroundMethodSelf-citationQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199 · cited 3×
“A domain-based structure was adopted similar to that used in other risk of bias tools such as the revised Cochrane Risk of Bias tool,(29) QUADAS-2,(36) ROBINSI(30) and ROBIS.”
InfluentialSelf-citationCochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,178
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · cited 11×
“A multivariable prediction model is defined as any combination or equation of two or more predictors for estimating the probability or risk for an individual.(7, 8, 33-35)”
InfluentialSelf-citation2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.005 · Cited by 4,945
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Circulation · doi:10.1161/01.cir.0000437741.48606.98 · Cited by 3,697
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Self-citationAssessing Bias in Studies of Prognostic Factors
Hayden, J. A., van der Windt, D. A., Cartwright, J. L., Côté, P., & Bombardier, C. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-4-201302190-00009 · Cited by 3,287 · cited 1×
“(25, 26) The QUIPS tool has been developed to assess the risk of bias in predictor finding (prognostic factor) studies.(28) The methodological quality of studies investigating the impact of a prediction model using a comparative randomised design can be assessed using the revised Cochrane risk of bias tool (ROB 2.”
MethodCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Self-citationSystematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
Cited by
Items 301–400 of 1,010
Scoping Review on the Diagnosis, Prognosis, and Treatment of Pediatric Disorders of Consciousness
Molteni, E., Canas, L. S., Briand, M., Estraneo, A., Font, C. C., Formisano, R., Валерьевна, Ф. Е., Gosseries, O., Howarth, R., Lanteri, P., et al. (2023) · Neurology · doi:10.1212/wnl.0000000000207473 · Cited by 45
Deimazar, G., & Sheikhtaheri, A. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105246 · Cited by 58
Tiruneh, S. A., Vu, T. T. T., Rolnik, D. L., Teede, H. J., & Enticott, J. (2024) · Current Hypertension Reports · doi:10.1007/s11906-024-01297-1 · Cited by 52
“The risk of bias (ROB) and concern for applicability [43] was assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST) tool by two authors (SAT and TV).”
MethodLiawrungrueang, W., Park, J.-B., Cholamjiak, W., Sarasombath, P., & Riew, K. D. (2024) · Global Spine Journal · doi:10.1177/21925682241274372 · Cited by 45
“8,9 The results showed Ayse Betul Oktay et al, 3 Zhongyi Han et al, 14 Kai-Uwe Lewandrowskl et al, 11 Alexandra Grob et al, 20 Jason Pui Yin Cheung et al, 5 Terence P. McSweeney et al, 21 and Wongthawat Liawrungrueang et al 4 were identi fi ed as having a low risk of bias and low concerns regarding…”
de Jong, Y., Ramspek, C. L., van der Endt, V. H. W., Rookmaaker, M. B., Blankestijn, P. J., Vernooij, R. W. M., Verhaar, M. C., Bos, W. J. W., Dekker, F. W., Ocak, G., et al. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.03.015 · Cited by 49
Machine Learning in the Prediction of Trauma Outcomes: A Systematic Review
Zhang, T., Nikouline, A., Lightfoot, D., & Nolan, B. (2022) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2022.05.011 · Cited by 45
Pan, Z., Zhang, R., Shen, S., Lin, Y., Zhang, L., Wang, X., Ye, Q., Wang, X., Chen, J., Zhao, Y., et al. (2023) · eBioMedicine · doi:10.1016/j.ebiom.2023.104443 · Cited by 43
Levkovich, I., & Omar, M. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02134-3 · Cited by 57
“For studies that were aimed at developing predictive tools using LLMs for the prevention of suicide attempts and that made no direct comparisons with existing methods, we utilized the prediction model Risk Of Bias Assessment Tool (PROBAST) [31].”
Kaiser, I., Pfahlberg, A. B., Uter, W. J. C., Heppt, M. V., Veierød, M. B., & Gefeller, O. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17217919 · Cited by 47
“Recently, the PROBAST tool has been developed through a consensus process to assess risk of bias and applicability of prediction model studies [89].”
Marcolino, M. S., Pires, M. C., Ramos, L. E. F., Silva, R. T., Oliveira, L. M., de Carvalho, R. L. R., Mourato, R. L. S., Sánchez‐Montalvá, A., Raventós, B., Anschau, F., et al. (2021) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2021.07.049 · Cited by 51
“…(TRIPOD) checklist and Prediction model Risk Of Bias AS- essment Tool (PROBAST) (Supplementary Material Tables S6 and 7) ( Moons et al., 2015 , Wolff et al., 2019 ). tudy subjects onsecutive adult patients (aged ≥ 18 years) with laboratoryonfirmed COVID-19 ( Organization, 2020 ) admitted to…”
InfluentialSingh, P., Adderley, N. J., Hazlehurst, J., Price, M. J., Tahrani, A. A., Nirantharakumar, K., & Bellary, S. (2021) · Diabetes Care · doi:10.2337/dc21-0166 · Cited by 50
Tousignant‐Laflamme, Y., Houle, C., Cook, C. E., Naye, F., LeBlanc, A., & Décary, S. (2022) · Physical Therapy · doi:10.1093/ptj/pzac023 · Cited by 47
“22,33,34 This step is essential to ensure that the model is of clinical value.”
Pecoraro, A., Campi, R., Bertolo, R. G., Mir, M. C., Marchioni, M., Serni, S., Joniau, S., Poppel, H. V., Albersen, M., & Roussel, E. (2023) · European Urology Oncology · doi:10.1016/j.euo.2022.11.007 · Cited by 49
Chen, Z., Liu, X., Yang, Q., Wang, Y.-J., Miao, K., Gong, Z., Yu, Y., Leonov, A., Liu, C., Feng, Z., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.1671 · Cited by 43
Considerations in the reliability and fairness audits of predictive models for advance care planning
Lu, J., Sattler, A., Wang, S., Khaki, A. R., Callahan, A., Fleming, S., Fong, R., Ehlert, B., Li, R. C., Shieh, L., et al. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.943768 · Cited by 40
“We then plotted the averaged predicted risk for each quintile on the x-axis and proportion of positive ground truth labels for each quintile on the y-axis (6,8–10,12 ,14 ).”
MethodBronchopulmonary dysplasia prediction models: a systematic review and meta-analysis with validation
Kwok, T. C., Batey, N., Luu, K. L., Prayle, A., & Sharkey, D. (2023) · Pediatric Research · doi:10.1038/s41390-022-02451-8 · Cited by 48
Burnett‐Bowie, S. M., Wright, N. C., Yu, E. W.-Y., Langsetmo, L., Yearwood, G. M. H., Crandall, C. J., Leslie, W. D., & Cauley, J. A. (2024) · Journal of Bone and Mineral Research · doi:10.1093/jbmr/zjae048 · Cited by 45
Cau, R., Pisu, F., Suri, J. S., & Saba, L. (2024) · European Journal of Radiology · doi:10.1016/j.ejrad.2024.111867 · Cited by 54
Issaiy, M., Zarei, D., Kolahi, S., & Liebeskind, D. S. (2024) · Journal of Neurology · doi:10.1007/s00415-024-12810-6 · Cited by 49
“The studies were assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST), designed for bias risk evaluation in four domains and the suitability of diagnostic and prognostic models [14].”
Artificial intelligence in risk prediction and diagnosis of vertebral fractures
Namireddy, S. R., Gill, S. S., Peerbhai, A., Kamath, A. G., Ramsay, D. S. C., Ponniah, H. S., Salih, A., Janković, D., Kalasauskas, D., Neuhoff, J., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-75628-2 · Cited by 50
Exploring Artificial Intelligence Biases in Predictive Models for Cancer Diagnosis
Smiley, A., Reátegui-Rivera, C. M., Villarreal‐Zegarra, D., Escobar-Agreda, S., & Finkelstein, J. (2025) · Cancers · doi:10.3390/cancers17030407 · Cited by 53
Mouselimis, D., Tsarouchas, A., Pagourelias, E., Bakogiannis, C., Theofilogiannakos, E. K., Loutradis, C., Fragakis, N., Vassilikos, V. P., & Papadopoulos, C. E. (2020) · Hellenic Journal of Cardiology · doi:10.1016/j.hjc.2020.04.008 · Cited by 45
Allotey, J., Laivuori, H., Snell, K. I., Smuk, M., Hooper, R., Chan, C. L., Ahmed, A., Chappell, L. C., von Dadelszen, P., Dodds, J., et al. (2020) · Health Technology Assessment · doi:10.3310/hta24720 · Cited by 30
Self-citationBarreñada, L., Ledger, A., Dhiman, P., Collins, G., Wynants, L., Verbakel, J. Y., Timmerman, D., Valentin, L., & Van Calster, B. (2024) · BMJ Medicine · doi:10.1136/bmjmed-2023-000817 · Cited by 42
Applications of Artificial Intelligence in Dental Medicine: A Critical Review
Sitaras, S., Tsolakis, I. A., Gelsini, M., Tsolakis, A. I., Schwendicke, F., Wolf, T. G., & Perlea, P. (2025) · International Dental Journal · doi:10.1016/j.identj.2024.11.009 · Cited by 42
Gao, Y., Cheng, S., Zhu, L., Wang, Q., Deng, W., Sun, Z., Wang, S., & Xue, H. (2022) · European Radiology · doi:10.1007/s00330-022-08922-0 · Cited by 48
Natural language processing for clinical notes in dentistry: A systematic review
Pethani, F., & Dunn, A. G. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104282 · Cited by 54
Democratizing Artificial Intelligence Imaging Analysis With Automated Machine Learning: Tutorial
Thirunavukarasu, A. J., Elangovan, K., Gutierrez, L., Li, Y., Tan, I., Keane, P. A., Korot, E., & Ting, D. S. W. (2023) · Journal of Medical Internet Research · doi:10.2196/49949 · Cited by 37
“While the curation process is identical to that described in steps 1A and 1B, there exists an additional option to collect data prospectively to facilitate a robust analysis of model performance in clinical conditions, with a lower risk of bias [34].”
Pan, Q., & Tong, M. (2024) · Renal Failure · doi:10.1080/0886022x.2024.2435483 · Cited by 48
Yao, X., Flórez, I. D., Zhang, P., Zhang, C., Zhang, Y., Wang, C., Liu, X., Nie, X., Wei, B., & Ghert, M. A. (2020) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12384 · Cited by 47
Länger, T., Favarato, M., Giudici, R., Bassi, G., Garberi, R., Villa, F., Gay, H., Zeduri, A., Bragagnolo, S., Molteni, A., et al. (2020) · Scandinavian Journal of Trauma Resuscitation and Emergency Medicine · doi:10.1186/s13049-020-00808-8 · Cited by 44
“01 1000 Random Soft Max D_FF_GNet [37] Langer et al.”
BackgroundPauwels, R. W. M., van der Woude, C. J., Nieboer, D., Steyerberg, E. W., Casanova, M. J., Gisbert, J. P., Kennedy, N. A., Lees, C. W., Louis, E. J., Molnár, T., et al. (2021) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2021.03.037 · Cited by 47
Development and reporting of artificial intelligence in osteoporosis management
Gatineau, G., Shevroja, E., Vendrami, C., Rodríguez, E. G., Leslie, W. D., Lamy, O., & Hans, D. B. (2024) · Journal of Bone and Mineral Research · doi:10.1093/jbmr/zjae131 · Cited by 50
“114 These checklists were used in combination to provide a general indicator of reporting standards, and to assess potential bias, highlighting the need for AI development and reporting standards that are reflective of the complex methods and unique terminologies.”
MethodMarouf, A. A., Rokne, J. G., & Alhajj, R. (2025) · Cancers · doi:10.3390/cancers17223638 · Cited by 43
“Unlike PROBAST, a tool for assessing the risk of bias and applicability of prediction models [48,49], there is no specific scoring to interpret the quality of a review by the AMSTAR 2.0 system.”
InfluentialYu, M.-Y., & Son, Y. (2024) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvae031 · Cited by 52
Prediction models for postoperative pulmonary complications: a systematic review and meta-analysis
Huang, Z., Han, Y., Zhuang, H., Jiang, J., Zhou, C., & Yu, H. (2025) · British Journal of Anaesthesia · doi:10.1016/j.bja.2025.04.025 · Cited by 48
Using Big Data and Predictive Analytics to Determine Patient Risk in Oncology
Parikh, R. B., Gdowski, A., Patt, D. A., Hertler, A., Mermel, C., & Bekelman, J. E. (2019) · American Society of Clinical Oncology Educational Book · doi:10.1200/edbk_238891 · Cited by 38
Tong‐Minh, K., Welten, I., Endeman, H., Hagenaars, T., Ramakers, C. R. B., Gommers, D. A. M. P. J., Gorp, E. C. M. V., & van der Does, Y. (2021) · BMC Emergency Medicine · doi:10.1186/s12873-021-00461-z · Cited by 42
Prediction of preterm birth using artificial intelligence: a systematic review
Akazawa, M., & Hashimoto, K. (2022) · Journal of Obstetrics and Gynaecology · doi:10.1080/01443615.2022.2056828 · Cited by 40
“Risk of bias was assessed by PROSBAST (prediction model risk of bias assessment tool) (Wolff et al. 2019).”
MethodBenavent, D., Carmona, L., Llorente, J. F. G., Montoro, M., Ramirez, S., Otón, T., Loza, E., & Gómez-Centeno, A. (2025) · Rheumatology International · doi:10.1007/s00296-025-05825-3 · Cited by 42
“In the supplementary material, we have included the excluded articles and the reasons for exclusion after full-text reading (Table S5).”
Examining Bias and Reporting in Oral Health Prediction Modeling Studies
Du, M., Haag, D. G., Song, Y., Lynch, J. W., & Mittinty, M. (2020) · Journal of Dental Research · doi:10.1177/0022034520903725 · Cited by 41
“Following PROBAST recommendations (Moons et al. 2019), we classified studies into high, low, and unclear ROB and recorded an answer of yes, probably yes, no, probably no, or no information for 20 signaling questions (Appendix Table 4) with supportive background (Appendix Table 5).”
BackgroundMethodInfluentialTufo, A., Pisano, M., Ansaloni, L., de Reuver, P., van Laarhoven, C. J. H. M., Davidson, B. R., & Gurusamy, K. S. (2020) · Journal of Laparoendoscopic & Advanced Surgical Techniques · doi:10.1089/lap.2020.0151 · Cited by 40
“20 The PROBAST tool has been mainly developed for risk prediction model.”
Zhang, K., Zhang, X., Ding, W., Xuan, N., Tian, B., Huang, T., Zhang, Z., Cui, W., Huang, H., & Zhang, G. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.699880 · Cited by 47
“Quality Assessment Two authors employed the PROBAST to assess the risk of bias and applicability concerns of the included studies (13).”
Chowdhury, M., Cervantes, E. G., Chan, W.-Y., & Seitz, D. P. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.738466 · Cited by 46
“The overall risk of bias (ROB) of the prediction models was judged as “low”, “high”, or “unclear”, and overall applicability of the prediction models was considered as “low concern”, “high concern”, and “unclear” according to the PROBAST checklist (11, 12).”
Barboi, C., Tzavelis, A., & Muhammad, L. N. (2022) · JMIR Medical Informatics · doi:10.2196/35293 · Cited by 37
“The ROB and applicability were assigned as low risk, high risk, or unclear risk according to PROBAST recommendations [42].”
MethodSuri, J. S., Agarwal, S., Jena, B., Saxena, S., El-Baz, A., Agarwal, V., Kalra, M. K., Saba, L., Viskovic, K., Fatemi, M., et al. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3174270 · Cited by 38
Goldberg, D., Mantero, A., Newcomb, C., Delgado, C., Forde, K. A., Kaplan, D. E., John, B., Nuchovich, N., Dominguez, B., Emanuel, E., et al. (2021) · Journal of Hepatology · doi:10.1016/j.jhep.2020.12.021 · Cited by 38
Bias Mitigation in Primary Health Care Artificial Intelligence Models: Scoping Review
Sasseville, M., Ouellet, S., Rhéaume, C., Sahlia, M., Couture, V., Després, P., Paquette, J.-S., Darmon, D., Bergeron, F., & Gagnon, M.-P. (2025) · Journal of Medical Internet Research · doi:10.2196/60269 · Cited by 38
Gokhale, S., Taylor, D., Gill, J., Hu, Y., Zeps, N., Lequertier, V., Prado, L., Teede, H. J., & Enticott, J. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1192969 · Cited by 48
“Studies at moderate-to-high ROB in these domains demonstrated unclear reporting of data source quality, availability of predictors during implementation, determination, definition, and consistency of outcomes, and inappropriate participant inclusion/exclusion.”
BackgroundMethodInfluentialReeve, K. A., Ön, B. I., Havla, J. B., Burns, J., Gosteli, M., Alabsawi, A., Alayash, Z., Götschi, A., Seibold, H., Mansmann, U., et al. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013606.pub2 · Cited by 59
Zhang, M., Zheng, Y., Maidaiti, X., Liang, B., Wei, Y., & Sun, F. (2024) · Health Data Science · doi:10.34133/hds.0165 · Cited by 44
Viljoen, J. L., Goossens, I., Monjazeb, S., Cochrane, D. M., Vargen, L. M., Jonnson, M. R., Blanchard, A. J. E., Li, S. M. Y., & Jackson, J. R. (2024) · Behavioral Sciences & the Law · doi:10.1002/bsl.2698 · Cited by 39
Prediction models for prostate cancer to be used in the primary care setting: a systematic review
Aladwani, M., Lophatananon, A., Ollier, W., & Muir, K. R. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-034661 · Cited by 40
Mari, T., Henderson, J., Maden, M., Nevitt, S. J., Duarte, R. V., & Fallon, N. B. (2021) · Journal of Pain · doi:10.1016/j.jpain.2021.07.011 · Cited by 41
van Boven, M. R., Henke, C. E., Leemhuis, A. G., Hoogendoorn, M., van Kaam, A. H., Königs, M., & Oosterlaan, J. (2022) · PEDIATRICS · doi:10.1542/peds.2021-056052 · Cited by 40
A Unified Framework on Generalizability of Clinical Prediction Models
Wan, B., Caffo, B. S., & Vedula, S. S. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.872720 · Cited by 41
“Several articles describe the objectives and design of studies to develop and validate CPMs for diagnosis and prognosis, their reporting, and risk of bias (Altman et al., 2009; Royston et al., 2009; Whiting et al., 2011; Moons et al., 2015; Wolff et al., 2019).”
BackgroundMethodClinical applications of artificial intelligence in liver imaging
Yamada, A., Kamagata, K., Hirata, K., Ito, R., Nakaura, T., Ueda, D., Fujita, S., Fushimi, Y., Fujima, N., Matsui, Y., et al. (2023) · La radiologia medica · doi:10.1007/s11547-023-01638-1 · Cited by 42
Better Reporting of Studies on Artificial Intelligence: CONSORT-AI and Beyond
Schwendicke, F., & Krois, J. (2021) · Journal of Dental Research · doi:10.1177/0022034521998337 · Cited by 37
“A range of methods and risk of bias tools for assessing diagnostic accuracy and prediction studies like QUADAS-2 (Whiting et al. 2011) and PROBAST (Wolff et al. 2019) can be employed to gauge bias.”
MethodSenior, M., Fanshawe, T. R., Fazel, M. S., & Fazel, S. (2021) · JCPP Advances · doi:10.1002/jcv2.12034 · Cited by 39
Russo, D., Mariani, P., Caponio, V. C. A., Russo, L. L., Fiorillo, L., Zhurakivska, K., Muzio, L. L., Laino, L., & Troiano, G. (2021) · Cancers · doi:10.3390/cancers13225755 · Cited by 42
“Domain four (analysis domain) is the one that most influenced the overall risk of bias [16,51].”
BackgroundMethodInfluentialComposite Outcomes in Clinical Prediction Modeling: Are We Trying to Predict Apples and Oranges?
Dash, K., Goodacre, S. W., & Sutton, L. (2022) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2022.01.046 · Cited by 41
Araújo, A. L. D., Moraes, M. C., Pérez-de-Oliveira, M. E., Silva, V. M. D., Saldivia-Siracusa, C., Pedroso, C. M., Lopes, M. A., Vargas, P. A., Kochanny, S., Pearson, A., et al. (2023) · Oral Oncology · doi:10.1016/j.oraloncology.2023.106386 · Cited by 36
Compte, R., Smith, I. G., Isaac, A., Danckert, N. P., McSweeney, T. P., Liantis, P., & Williams, F. M. K. (2023) · European Spine Journal · doi:10.1007/s00586-023-07718-0 · Cited by 38
“Articles were tabled according to the PROBAST tool [43] to summarize the design of each study, assess risk of bias and determine the applicability of included models (S10).”
MethodMishra, A. K., Chong, B., Arunachalam, S. P., Oberg, A. L., & Majumder, S. (2024) · The American Journal of Gastroenterology · doi:10.14309/ajg.0000000000002870 · Cited by 42
Ladin, K., Cuddeback, J. K., Duru, O. K., Goel, S., Harvey, W., Park, J. G., Paulus, J. K., Sackey, J., Sharp, R. R., Steyerberg, E. W., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01245-y · Cited by 41
“…” models to ensure accurate predictions across racial categories (subgroup validity), and tocomplementotherexistingguidelines,suchastheTransparentReporting ofamultivariablepredictionmodelforIndividualPrognosisOrDiagnosis (TRIPOD) 58 and Prediction model Risk Of Bias ASsessment Tool (PROBAST) 63 .”
MethodLiu, Y., Shen, X., Zhang, Y., Wang, Z., Tian, Y., Dai, J., & Cao, Y. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00850-8 · Cited by 49
“To rigorously assess the risk of methodological bias, this review applied the prediction model risk of bias assessment tool (PROBAST) [34].”
Perry, B. I., Upthegrove, R., Crawford, O., Jang, S.-H., Lau, E., McGill, I., Carver, E., Jones, P. B., & Khandaker, G. M. (2020) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.13212 · Cited by 37
“%) either at unclear or high risk of bias following assessment with the PROBAST tool (14).”
BackgroundMethodInfluentialRisk assessments for violence and reoffending: Implementation and impact on risk management.
Viljoen, J. L., & Vincent, G. M. (2020) · Clinical Psychology Science and Practice · doi:10.1111/cpsp.12378 · Cited by 40
Machine learning for the life-time risk prediction of Alzheimer’s disease: a systematic review
Rowe, T. W., Katzourou, I. K., Stevenson‐Hoare, J. O., Bracher‐Smith, M., Ivanov, D. K., & Escott‐Price, V. (2021) · Brain Communications · doi:10.1093/braincomms/fcab246 · Cited by 44
“The risk of bias (ROB) was assessed by using the prediction model risk of bias assessment tool (PROBAST).(20)”
Intelligent Clinical Decision Support
Pinsky, M. R., Dubrawski, A. W., & Clermont, G. (2022) · Sensors · doi:10.3390/s22041408 · Cited by 44
External Validation of e‐ASPECTS Software for Interpreting Brain CT in Stroke
Mair, G., White, P., Bath, P. M., Muir, K. W., Salman, R. A., Martin, C., Dye, D., Chappell, F. M., Vacek, A., von Kummer, R., et al. (2022) · Annals of Neurology · doi:10.1002/ana.26495 · Cited by 38
“We used the Prediction model Risk Of Bias ASsessment Tool (PROBAST) to assess the risk of bias and applicability of our testing.(20) To aid understanding, we summarized results as proportions with or without expert agreement per 100 patients.”
MethodAmacher, S. A., Blatter, R., Briel, M., Appenzeller‐Herzog, C., Bohren, C., Becker, C., Beck, K., Groß, S., Tisljar, K., Sutter, R., et al. (2022) · Critical Care · doi:10.1186/s13054-022-04263-y · Cited by 36
“Moons KGM, Wolff RF, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S. PROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration.”
BackgroundMethodInfluentialFardous, M., & Hasan, M. (2021) · doi:10.63125/3f0mwc90 · Cited by 36
Kom, I. Y., Dongelmans, D. A., de Keizer, N. F., Jager, K. J., Schut, M. C., Abu–Hanna, A., & Klopotowska, J. E. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad014 · Cited by 36
“80 However, most of the PROBAST items are also covered by the CHARMS checklist, albeit without formal judgments on risk of bias.”
BackgroundMachine learning‐based prediction models for pressure injury: A systematic review and meta‐analysis
Pei, J., Guo, X., Tao, H., Wei, Y., Zhang, H., Ma, Y., & Han, L. (2023) · International Wound Journal · doi:10.1111/iwj.14280 · Cited by 41
“19 PROBAST assessment included the following four areas: participants, predictors, outcome and analysis.”
Zu, W., Huang, X., Xu, T., Du, L., Wang, Y., Wang, L., & Nie, W. (2023) · PLoS ONE · doi:10.1371/journal.pone.0287308 · Cited by 40
“PROBAST (Prediction model Risk Of Bias ASsessment Tool) was used to guide the risk of bias assessment in this review [15].”
BackgroundMethodInfluentialDevelopment of a Health Equity Framework for the US Preventive Services Task Force
Lin, J. S., Webber, E. M., Bean, S., & Evans, C. V. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.1875 · Cited by 38
Wang, J., Tozzi, F., Ashraf‐Ganjouei, A., Romero‐Hernández, F., Feng, J., Calthorpe, L. M., Castro, M., Davis, G. L., Withers, J., Zhou, C. J., et al. (2024) · Journal of Gastrointestinal Surgery · doi:10.1016/j.gassur.2024.03.006 · Cited by 38
Hama, T., Alsaleh, M. M., Allery, F., Choi, J. W., Tomlinson, C., Wu, H., Lai, A. G., Pontikos, N., & Thygesen, J. H. (2025) · Journal of Medical Internet Research · doi:10.2196/57358 · Cited by 43
“The Prediction Model Study Risk of Bias Assessment Tool (PROBAST) [20] was used to evaluate both the risk of bias (ROB) and the applicability of the best-performing DL models in the included studies.”
Gomes, C., Geels, J., Debray, T. P. A., Malekzadeh, A., Asselbergs, F. W., & Linschoten, M. (2025) · BMJ · doi:10.1136/bmj-2025-084062 · Cited by 40
“The Prediction model Risk Of Bias ASsessment Tool (PROBAST) was used to evaluate the risk of bias of retrieved studies, 24 according to the four key domains: participants, predictors, outcome, and analysis.”
Risk models and scores for metabolic syndrome: systematic review protocol
Ibrahim, M. S., Pang, D., Randhawa, G., & Pappas, Y. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-027326 · Cited by 39
Cai, C., Runte, J., Ostrer, I. R., Berry, K., Ponce, N. A., Rodríguez, M., Bertozzi, S., White, J. S., & Kahn, J. G. (2020) · PLoS Medicine · doi:10.1371/journal.pmed.1003013 · Cited by 35
Development of frailty measurement tools using administrative health data: A systematic review
Nghiem, S. H., Sajeewani, D., Henderson, K., Afoakwah, C., Byrnes, J., Moyle, W., & Scuffham, P. (2020) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2020.104102 · Cited by 38
Ophey, A., Roheger, M., Folkerts, A., Skoetz, N., & Kalbe, E. (2020) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2020.575804 · Cited by 40
“Instead of using two different risk of bias assessment tools [QUIPS (Hayden et al., 2013) for prognostic factor finding studies and Prediction Model Risk of Bias Assessment Tool (Wolff et al., 2019) for prognostic model studies], risk of bias of both prognostic factor finding and prognostic model studies was assessed…”
Prognostic models for predicting relapse or recurrence of major depressive disorder in adults
Moriarty, A. S., Meader, N., Snell, K. I., Riley, R. D., Paton, L. W., Chew-Graham, C. A., Gilbody, S., Churchill, R., Phillips, R. S., Ali, S., et al. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013491.pub2 · Cited by 36
Self-citationPrediction models for severe manifestations and mortality due to COVID ‐19: A systematic review
Miller, J. L., Tada, M., Goto, M., Chen, H., Dang, E., Mohr, N. M., & Lee, S. (2022) · Academic Emergency Medicine · doi:10.1111/acem.14447 · Cited by 41
Duong, H. P., Garcia, A., Hilfiker, R., Léger, B., & Lüthi, F. (2022) · Frontiers in Rehabilitation Sciences · doi:10.3389/fresc.2021.791351 · Cited by 39
Rahimi, A. K., Canfell, O. J., Chan, W., Sly, B. P., Pole, J. D., Sullivan, C. M., & Shrapnel, S. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104758 · Cited by 41
Predicting postpartum haemorrhage: A systematic review of prognostic models
Carr, B. L., Jahangirifar, M., Nicholson, A. E., Li, W., Mol, B. W., & Licqurish, S. (2022) · Australian and New Zealand Journal of Obstetrics and Gynaecology · doi:10.1111/ajo.13599 · Cited by 40
MRI Radiomics and Predictive Models in Assessing Ischemic Stroke Outcome—A Systematic Review
Dragoș, H.-M., Stan, A. D., Pintican, R. M., Feier, D. S., Lebovici, A., Panaitescu, P.-Ș., Dina, C., Strilciuc, Ș., & Muresanu, D. F. (2023) · Diagnostics · doi:10.3390/diagnostics13050857 · Cited by 37
“PROBAST [31,32] was designed for use in systematic review or prediction model studies and consists of four domains (participants, predictors, outcome and analysis) containing twenty signaling questions to facilitate risk of bias assessment.”
MethodInfluentialWang, J., Yin, M., & Wen, H. C. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02247-8 · Cited by 40
“Risk of bias assessment Two investigators used the prediction model risk of bias(ROB) assessment tool (PROBAST) to assess the ROB of each model in each study and the applicability of our reviewed questions [13].”
MethodPandey, A. K., Okaj, I., Ichhpuniani, S., Tao, B. K.-L., Kaur, H., Spence, J. D., Young, J., Healey, J. S., Devereaux, P. J., Um, K. J., et al. (2023) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2023.08.161 · Cited by 34
Xu, W., Zhou, Y., Jiang, Q., Fang, Y., & Yang, Q. (2024) · Frontiers in Endocrinology · doi:10.3389/fendo.2024.1407348 · Cited by 34
Acute Kidney Injury Prognosis Prediction Using Machine Learning Methods: A Systematic Review
Lin, Y., Shi, T., & Kong, G. (2024) · Kidney Medicine · doi:10.1016/j.xkme.2024.100936 · Cited by 34
Reilly, J. R., Gabbe, B. J., Brown, W. A., Hodgson, C. L., & Myles, P. S. (2020) · ANZ Journal of Surgery · doi:10.1111/ans.16255 · Cited by 37
“Risk of bias assessment was performed on these tools using the Prediction Model Risk of Bias Assessment Tool.15 Properties analysed include predictive performance (discrimination and calibration), number and type of variables in the model or score, size of development and validation population relative to number of va…”
MethodPrognostic models for amyotrophic lateral sclerosis: a systematic review
Xu, L., He, B., Zhang, Y., Chen, L., Fan, D., Zhan, S., & Wang, S. (2021) · Journal of Neurology · doi:10.1007/s00415-021-10508-7 · Cited by 35
Prognostic Models Predicting Mortality in Preterm Infants: Systematic Review and Meta-analysis
van Beek, P. E., Andriessen, P., Onland, W., & Schuit, E. (2021) · PEDIATRICS · doi:10.1542/peds.2020-020461 · Cited by 34
Alhmoud, B., Bonnici, T., Patel, R. S., Melley, D. D., Williams, B., & Banerjee, A. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-045849 · Cited by 41
Buchlak, Q. D., Esmaili, N., Bennett, C. C., & Farrokhi, F. R. (2021) · Acta neurochirurgica. Supplementum · doi:10.1007/978-3-030-85292-4_32 · Cited by 39
Driggs, D., Selby, I., Roberts, M., Gkrania‐Klotsas, E., Rudd, J. H. F., Yang, G., Babar, J. L., Sala, E., & Schönlieb, C. (2021) · Radiology Artificial Intelligence · doi:10.1148/ryai.2021210011 · Cited by 37
“For example, the Prediction model Risk Of Bias Assessment Tool (PROBAST) checklist (14) for assessing the risk of bias in medical models requires models to be validated on an external dataset, but in machine learning research, it is common practice to validate a model using an 80:20 training-to-testing split from the…”
Method
Metadata
- Authors
- Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group†
- Affiliations
- Kleijnen Systematic Reviews (United Kingdom) (GB)
- Utrecht University (NL)
- Keele University (GB)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom (G.S.C.)
- Maastricht University (NL)
- Institute of Applied Health Research, National Institute for Health Research Birmingham Biomedical Research Centre, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom (S.M.)
- ORCID
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2019-01-01
- Volume
- 170
- Issue
- 1
- Pages
- 51-58
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W2907554860
- Cited by
- 2,984
- References
- 42
- Citations by year
- 2026: 921 · 2025: 561 · 2024: 416 · 2023: 343 · 2022: 303 · 2021: 250 · 2020: 140 · 2019: 45
- FWCI
- 83.39
- Type
- journal article
- Language
- en
- Open access
- Licence
- other-oa
- www.acpjournals.org/journal/aim/text-and-data-mining (tdm)
- Locations
- Annals of Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- Bristol Research (University of Bristol) (acceptedVersion, other-oa)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- FreiDok plus (Universitätsbibliothek Freiburg) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- Radboud Repository (Radboud University) (submittedVersion)
- Explore Bristol Research (submittedVersion, other-oa)
- Apollo (University of Cambridge) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Apollo
- Funders
- National Institute for Health Research Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (ZONMW 918.10.615, ZONMW 912.08.004)
- National Institute for Health Research School for Primary Care Research (390)
- National Institute for Health Research Biomedical Research Centre, Oxford
- National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) West at University Hospitals Bristol NHS Foundation Trust
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- PROBAST, risk of bias assessment, diagnostic model, prognostic model, applicability, critical appraisal, reporting guidelines, Delphi consensus, systematic reviews, personalized prediction
- MeSH terms
- Systematic Reviews as Topic, Decision Support Techniques, Delphi Technique, Diagnosis, Humans, Prognosis, Research Design, Models, Statistical, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.