PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Clinical prediction models combine multiple predictors to estimate risk for the presence of a particular condition (diagnostic models) or the occurrence of a certain event in the future (prognostic models). PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool for assessing the risk of bias (ROB) and applicability of diagnostic and prognostic prediction model studies, was developed by a steering group that considered existing ROB tools and reporting guidelines. The tool was informed by a Delphi procedure involving 38 experts and was refined through piloting. PROBAST is organized into the following 4 domains: participants, predictors, outcome, and analysis. These domains contain a total of 20 signaling questions to facilitate structured judgment of ROB, which was defined to occur when shortcomings in study design, conduct, or analysis lead to systematically distorted estimates of model predictive performance. PROBAST enables a focused and transparent approach to assessing the ROB and applicability of studies that develop, validate, or update prediction models for individualized predictions. Although PROBAST was designed for systematic reviews, it can be used more generally in critical appraisal of prediction model studies. Potential users include organizations supporting decision making, researchers and clinicians who are interested in evidence-based medicine or involved in guideline development, journal editors, and manuscript reviewers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329 · cited 3×
“A domain-based structure was adopted similar to that used in other risk of bias tools such as the revised Cochrane Risk of Bias tool,(29) QUADAS-2,(36) ROBINSI(30) and ROBIS.”
BackgroundMethodSelf-citationQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199 · cited 3×
“A domain-based structure was adopted similar to that used in other risk of bias tools such as the revised Cochrane Risk of Bias tool,(29) QUADAS-2,(36) ROBINSI(30) and ROBIS.”
InfluentialSelf-citationCochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,178
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · cited 11×
“A multivariable prediction model is defined as any combination or equation of two or more predictors for estimating the probability or risk for an individual.(7, 8, 33-35)”
InfluentialSelf-citation2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.005 · Cited by 4,945
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Circulation · doi:10.1161/01.cir.0000437741.48606.98 · Cited by 3,697
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Self-citationAssessing Bias in Studies of Prognostic Factors
Hayden, J. A., van der Windt, D. A., Cartwright, J. L., Côté, P., & Bombardier, C. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-4-201302190-00009 · Cited by 3,287 · cited 1×
“(25, 26) The QUIPS tool has been developed to assess the risk of bias in predictor finding (prognostic factor) studies.(28) The methodological quality of studies investigating the impact of a prediction model using a comparative randomised design can be assessed using the revised Cochrane risk of bias tool (ROB 2.”
MethodCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Self-citationSystematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
Cited by
Items 201–300 of 1,010
Machine Learning in the Diagnosis and Prognostic Prediction of Dental Caries: A Systematic Review
Reyes, L. T., Knorst, J. K., Ortiz, F. R., & Ardenghi, T. M. (2022) · Caries Research · doi:10.1159/000524167 · Cited by 73
“The quality and applicability of prediction studies were measured on the prediction model risk of bias assessment tool [Wolff et al., 2019].”
MethodCai, S., Li, J., Gao, J., Pan, W., & Zhang, Y. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2022.104340 · Cited by 71
Liu, M., Yang, X., Chen, G., Ding, Y., Shi, M., Sun, L., Huang, Z., Liu, J., Liu, T., Yan, R., et al. (2022) · Frontiers in Physiology · doi:10.3389/fphys.2022.896969 · Cited by 71
“This is part of the quality assessment that we follow from the Predictive Model Deviation Risk Assessment Tool (PROBAST) (Wolff et al., 2019).”
MethodHsu, W. W. Y., Hippe, D. S., Nakhaei, N., Wang, P., Zhu, B., Siu, N., Ahsen, M. E., Lotter, W. E., Sorensen, A. G., Naeim, A., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.42343 · Cited by 66
Klement, W., & El Emam, K. (2023) · Journal of Medical Internet Research · doi:10.2196/48763 · Cited by 59
“[3,4,27,45,46,50,52-59,61 ,62 ,79 In this paper, we developed a consolidated list of 37 reporting items for ML prognostic and diagnostic modeling studies conducted in silico or in shadow mode.”
den Bosch, L. V., Schuit, E., van der Laan, H. P., Reitsma, J. B., Moons, K. G. M., Steenbakkers, R. J. H. M., Hoebers, F. J. P., Langendijk, J. A., & van der Schaaf, A. (2020) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2020.04.012 · Cited by 70
“Even with an adequate model development strategy, many other factors influence the quality and performance of NTCPmodels [24].”
BackgroundMethodSelf-citationKareemi, H., Vaillancourt, C., Rosenberg, H., Fournier, K., & Yadav, K. (2020) · Academic Emergency Medicine · doi:10.1111/acem.14190 · Cited by 66
“However, these thresholds often have unsubstantiated clinical relevance, thus introducing bias.(20) The so-called “threshold-free” measures of discrimination and calibration are therefore preferred for overall prediction model assessment.”
BackgroundMethodVenema, E., Wessler, B. S., Paulus, J. K., Salah, R., Raman, G., Leung, L. Y., Koethe, B. C., Nelson, J., Park, J. G., van Klaveren, D., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.017 · Cited by 65
Slieker, R. C., van der Heijden, A. A. W. A., Siddiqui, M. K., Langendoen‐Gort, M., Nijpels, G., Herings, R. M. C., Feenstra, T., Moons, K. G. M., Bell, S., Elders, P. J. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2134 · Cited by 70
Self-citationRisk Prediction Models for Kidney Cancer: A Systematic Review
Harrison, H., Thompson, R. E., Lin, Z., Rossi, S. H., Stewart, G. D., Griffin, S. J., & Usher-Smith, J. A. (2021) · European Urology Focus · doi:10.1016/j.euf.2020.06.024 · Cited by 58
“Additionally, each model was evaluated by one reviewer (H.H.) using the PROBAST assessment tool [22,23].”
Prediction models for depression risk among older adults: systematic review and critical appraisal
Tan, J., Ma, C., Zhu, C., Wang, Y., Zou, X., Li, H., Li, J., He, Y., & Wu, C. (2022) · Ageing Research Reviews · doi:10.1016/j.arr.2022.101803 · Cited by 70
Prognostic models for mortality risk in patients requiring ECMO
Pladet, L. C. A., Barten, J. M. M., Vernooij, L. M., Kraemer, C. V. E., Bunge, J. J. H., Scholten, E., Montenij, L. J., Kuijpers, M., Donker, D. W., Cremer, O. L., et al. (2023) · Intensive Care Medicine · doi:10.1007/s00134-022-06947-z · Cited by 66
“The Prediction model Risk Of BiAS Tool (PROBAST) was used to assess the risk of bias and concern for applicability per model, using signaling questions in four different domains (participants, predictors, outcomes and analysis) [18].”
BackgroundMethodDebray, T. P. A., Collins, G. S., Riley, R. D., Snell, K. I. E., Van Calster, B., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071058 · Cited by 57
“113 114 Use of this tool involves assessment of four domains to cover key aspects of prediction model studies: participants, predictors, outcome, and analysis.”
MethodSelf-citationWang, H. E., Landers, M., Adams, R., Subbaswamy, A., Kharrazi, H., Gaskin, D. J., & Saria, S. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac065 · Cited by 63
Kabore, E., MacDonald, C., Kaboré, A., Didier, R. J., Arveux, P., Méda, N., Boutron‐Ruault, M., & Guénancia, C. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.0569 · Cited by 70
Vernooij, J. E. M., Koning, N. J., Geurts, J. W., Holewijn, S., Preckel, B., Kalkman, C. J., & Vernooij, L. M. (2023) · Anaesthesia · doi:10.1111/anae.15988 · Cited by 65
“Figure 3 Risk of bias and applicability of pre-operative 30-daymortality riskmodels with PROBAST [24, 70].”
MethodSidey-Gibbons, C., Pfob, A., Asaad, M., Boukovalas, S., Lin, Y.-L., Selber, J. C., Butler, C. E., & Offodile, A. C. (2021) · JCO Clinical Cancer Informatics · doi:10.1200/cci.20.00088 · Cited by 60
“We used the Prediction Model Study Risk of Bias Assessment Tool (PROBAST) to ensure high accuracy when these models were used elsewhere.(25) We normalized the range of all values using feature scaling.”
Aquino, Y. S. J., Carter, S. M., Houssami, N., Braunack-Mayer, A., Win, K. T., Degeling, C., Wang, L., & Rogers, W. A. (2025) · Journal of Medical Ethics · doi:10.1136/jme-2022-108850 · Cited by 62
“18 These solutions are yet to be widely implemented.”
Nguyen, N. H., Picetti, D., Dulai, P. S., Jairath, V., Sandborn, W. J., Ohno-Machado, L., Chen, P. L., & Singh, S. (2021) · Journal of Crohn s and Colitis · doi:10.1093/ecco-jcc/jjab155 · Cited by 68
Corti, C., Cobanaj, M., Marian, F., Dee, E. C., Lloyd, M. R., Marcu, S., Dombrovschi, A., Biondetti, G. P., Batalini, F., Celi, L. A., et al. (2022) · Cancer Treatment Reviews · doi:10.1016/j.ctrv.2022.102410 · Cited by 67
van Royen, F. S.-A., Moons, K. G. M., Geersing, G., & van Smeden, M. (2022) · European Respiratory Journal · doi:10.1183/13993003.00250-2022 · Cited by 68
Self-citationVasey, B., Novak, A., Ather, S., Ibrahim, M., & McCulloch, P. G. (2023) · Clinical Radiology · doi:10.1016/j.crad.2022.09.131 · Cited by 74
Abd-Alrazaq, A. A., Alajlani, M., Ahmad, R., AlSaad, R., Aziz, S., Ahmed, A., Alsahli, M. A., Damseh, R., & Sheikh, J. I. (2023) · Journal of Medical Internet Research · doi:10.2196/52622 · Cited by 80
“We made adjustments by replacing certain nonapplicable criteria with more relevant ones from the Prediction Model Risk of Bias Assessment Tool [38].”
Cao, Y., Dai, J., Wang, Z., Zhang, Y., Shen, X., Liu, Y., & Tian, Y. (2025) · Journal of Behavioral Data Science · doi:10.35566/jbds/caoyc · Cited by 75
“The risk of bias in the studies included in this systematic review was assessed using the Prediction model Risk Of Bias Assessment Tool (PROBAST, Wolff et al., 2019).”
Tewarie, I. A., Senders, J. T., Kremer, S. W. F., Devi, S., Gormley, W. B., Arnaout, O. M., Smith, T. R., & Broekman, M. L. D. (2020) · Neurosurgical Review · doi:10.1007/s10143-020-01430-z · Cited by 64
Kremers, F. C. C., Venema, E., Duvekot, M. H. C., Yo, L. S. F., Bokkers, R. P. H., Nijeholt, G. J. L. À., van Es, A., van der Lugt, A., Majoie, C. B. L. M., Burke, J., et al. (2021) · Stroke · doi:10.1161/strokeaha.120.033445 · Cited by 66
Virdee, P. S., Marian, I. R., Mansouri, A., Elhussein, L., Kirtley, S., Holt, T., & Birks, J. (2020) · Cancers · doi:10.3390/cancers12092348 · Cited by 63
“Goshen 2017 [28] developed a prediction model for men and women separately and under the recommendation of the Cochrane PROBAST study group [12], both models scored high risk of bias in all four domains because neither underwent internal or external validation.”
MethodHalligan, S., Menu, Y., & Mallett, S. (2021) · European Radiology · doi:10.1007/s00330-021-07971-1 · Cited by 63
Self-citationKolk, M. Z. H., Deb, B., Ruipérez-Campillo, S., Bhatia, N. K., Clopton, P., Wilde, A. A. M., Narayan, S. M., Knops, R. E., & Tjong, F. V. Y. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104462 · Cited by 67
“21 All studies were scored on risk of bias for four categories (i.e. participants, predictors, outcome, and analysis).”
BackgroundMohammadi, I., Firouzabadi, S. R., Hosseinpour, M., Akhlaghpasand, M., Hajikarimloo, B., Tavanaei, R., Izadi, A., Nejad, S. Z., & Eghbali, F. (2024) · Journal of Translational Medicine · doi:10.1186/s12967-024-05481-4 · Cited by 69
“Quality assessment of non-HPI studies was conducted using the PROBAST tool [14], while quality assessment of studies utilizing HPI was done using the Jadad scale [15] or Newcastle-Ottowa scale [16] for controlled trials and observational studies respectively.”
MethodExploring the efficacy and potential of large language models for depression: A systematic review
Omar, M., & Levkovich, I. (2024) · Journal of Affective Disorders · doi:10.1016/j.jad.2024.11.052 · Cited by 67
Quiroz, J. C., Feng, Y.-Z., Cheng, Z.-Y., Rezazadegan, D., Chen, P.-K., Lin, Q.-T., Qian, L., Liu, X.-F., Berkovsky, S., Coiera, E., et al. (2021) · JMIR Medical Informatics · doi:10.2196/24572 · Cited by 52
Predicting lung cancer survival based on clinical data using machine learning: A review
Altuhaifa, F. A., Win, K. T., & Su, G. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107338 · Cited by 69
Strijker, M., Chen, J., Mungroop, T. H., Jamieson, N. B., van Eijck, C. H. J., Steyerberg, E. W., Wilmink, J. W., Koerkamp, B. G., van Laarhoven, H. W. M., & Besselink, M. G. (2019) · British journal of surgery · doi:10.1002/bjs.11111 · Cited by 61
ICU Delirium-Prediction Models: A Systematic Review
Ruppert, M. M., Lipori, J., Patel, S. H., Ingersent, E., Cupka, J., Ozrazgat‐Baslanti, T., Loftus, T. J., Rashidi, P., & Bihorac, A. (2020) · Critical Care Explorations · doi:10.1097/cce.0000000000000296 · Cited by 76
Ballanti, S., Campagnini, S., Liuzzi, P., Hakiki, B., Scarpino, M., Macchi, C., Oddo, C. M., Carrozza, M. C., Grippo, A., & Mannini, A. (2022) · Clinical Neurophysiology · doi:10.1016/j.clinph.2022.09.017 · Cited by 64
Machine Learning Augmented Interpretation of Chest X-rays: A Systematic Review
Ahmad, H. K., Milne, M., Buchlak, Q. D., Ektas, N., Sanderson, G., Chamtie, H., Karunasena, S., Chiang, J., Holt, X., Tang, C. H. M., et al. (2023) · Diagnostics · doi:10.3390/diagnostics13040743 · Cited by 67
“The PROBAST [40] ROB tool assessed shortcomings in study design, conduct, and analysis that may have put the results of a study at risk of being flawed or biased.”
InfluentialLiu, Z., Khuong, J. N., Caruana, C. B., Jackson, S. M., Campbell, R. J., Ramson, D. M., Penny‐Dimri, J. C., Kluger, M., Segal, R., & Perry, L. A. (2019) · Heart Lung and Circulation · doi:10.1016/j.hlc.2019.11.021 · Cited by 58
TRIPOD Reporting Guidelines for Diagnostic and Prognostic Studies
Patzer, R. E., Kaji, A. H., & Fong, Y. (2021) · JAMA Surgery · doi:10.1001/jamasurg.2021.0537 · Cited by 58
Perry, L. A., Liu, Z., Loth, J., Penny‐Dimri, J. C., Plummer, M. P., Segal, R., & Smith, J. A. (2021) · Journal of Cardiothoracic and Vascular Anesthesia · doi:10.1053/j.jvca.2021.07.001 · Cited by 63
Prediction modelling in the early detection of neonatal sepsis
Sahu, P., Stanly, E. A. R., Lewis, L., Prabhu, K., Rao, M., & Kunhikatta, V. (2022) · World Journal of Pediatrics · doi:10.1007/s12519-021-00505-1 · Cited by 60
“Partly, the studies (five studies) had applied split validation (70:30 ratio) concepts for the derivation and validation of the model, while one of the studies had applied external validation [21]; while the other four studies had utilised bootstrap or concordance statistics methodology for internal validation.”
BackgroundMethodInfluentialTorres‐Navarro, I., Briz‐Redón, Á., & Estrada, R. B. (2019) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.16137 · Cited by 56
Applications of natural language processing at emergency department triage: A narrative review
Stewart, J. E., Lü, J., Goudie, A., Arendts, G., Meka, S., Freeman, S., Walker, K., Sprivulis, P., Sanfilippo, F. M., Bennamoun, M., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0279953 · Cited by 70
“Risk of bias in studies was assessed independently by two authors (JS and JL) using the Prediction model Risk of Bias Assessment Tool (PROBAST) [33].”
Christou, C. D., & Tsoulfas, G. (2021) · World Journal of Gastroenterology · doi:10.3748/wjg.v27.i37.6191 · Cited by 54
Sun, Z., Dong, W., Shi, H., Ma, H., Cheng, L., & Huang, Z. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.812276 · Cited by 67
Machine Learning–Based Prediction Models for Delirium: A Systematic Review and Meta-Analysis
Xie, Q., Wang, X., Pei, J., Wu, Y., Guo, Q., Su, Y., Yan, H., Nan, R., Chen, H., & Dou, X. (2022) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2022.06.020 · Cited by 58
Lu, J. H., Callahan, A., Patel, B. S., Morse, K. E., Dash, D., Pfeffer, M. A., & Shah, N. H. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.27779 · Cited by 52
O’Neill, J. E. R., Corish, C. A., & Horner, K. M. (2023) · Sports Medicine · doi:10.1007/s40279-023-01896-z · Cited by 59
Chiasakul, T., Lam, B. D., McNichol, M. C., Robertson, W. E., Rosovsky, R. P. G., Lake, L. L., Vlachos, I. S., Adamski, A., Reyes, N., Abe, K., et al. (2023) · European Journal Of Haematology · doi:10.1111/ejh.14110 · Cited by 57
Artificial intelligence in osteoarthritis detection: A systematic review and meta-analysis
Mohammadi, S., Salehi, M. A., Jahanshahi, A., Farahani, M. S., Zakavi, S. S., Behrouzieh, S., Gouravani, M., & Guermazi, A. (2023) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2023.09.011 · Cited by 59
Cai, Y., Gong, D.-X., Tang, L.-Y., Cai, Y. H., Li, H., Jing, T.-C., Gong, M., Hu, W., Zhang, Z.-W., Zhang, X., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/47645 · Cited by 68
“Ultimately, 31 papers were included in the preliminary summary of candidate items (Multimedia Appendix 2 [25,27, 31-60 31-60 ]).”
Kretzer, S., Lawrence, A. J., Pollard, R., Ma, X., Chen, P. J., Amasi-Hartoonian, N., Pariante, C. M., Vallée, C., Meaney, M., & Dazzan, P. (2024) · Biological Psychiatry · doi:10.1016/j.biopsych.2024.06.012 · Cited by 58
Moving towards the use of artificial intelligence in pain management
Antel, R., Whitelaw, S., Gore, G. C., & Ingelmo, P. M. (2024) · European Journal of Pain · doi:10.1002/ejp.4748 · Cited by 59
Machine Learning Operations in Health Care: A Scoping Review
Rajagopal, A., Ayanian, S., Ryu, A. J., Qian, R., Legler, S. R., Peeler, E. A., Issa, M., Coons, T. J., & Kawamoto, K. (2024) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2024.06.009 · Cited by 52
“59,60 Although several evaluation frameworks have been developed in recent years, 61-63 they are mainly centered around the reporting of clinical AI studies as opposed to real-world operational guidelines.”
Artificial Intelligence and Machine Learning in Cancer Pain: A Systematic Review
Salama, V., Godinich, B. M., Geng, Y., Humbert‐Vidan, L., Maule, L., Wahid, K. A., Naser, M. A., He, R., Mohamed, A., Fuller, C. D., et al. (2024) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2024.07.025 · Cited by 58
Externally validated prediction models for pre‐eclampsia: systematic review and meta‐analysis
Tiruneh, S. A., Vu, T. T. T., Moran, L., Callander, E. J., Allotey, J., Thangaratinam, S., Rolnik, D. L., Teede, H. J., Wang, R., & Enticott, J. (2024) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.27490 · Cited by 62
“We extracted data using the CHARMS checklist and appraised the risk of bias using the PROBAST tool.”
MethodFrondelius, T., Atkova, I., Miettunen, J., Rello, J., Vesty, G. M., Chew, H. S. J., & Jansson, M. M. (2023) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2023.11.009 · Cited by 58
Li, W. W., Heward, C., Merrick, A., Astridge, B., & Leow, T. Y. Q. (2024) · Journal of Pacific Rim Psychology · doi:10.1177/18344909241252325 · Cited by 53
“The Prediction Model Study Risk of Bias Assessment Tool (PROBAST) was employed to assess the risk of bias and concerns with regards to the applicability of the included studies (Wolff et al., 2019).”
Prediction models for determining the success of labor induction: A systematic review
Meier, K., Parrish, J., & D'Souza, R. D. (2019) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1111/aogs.13589 · Cited by 53
A novel nomogram to predict early neurological deterioration in patients with acute ischaemic stroke
Gong, P., Zhang, X., Gong, Y., Liu, Y., Wang, S., Li, Z., Chen, W., Zhou, F., Zhou, J., Jiang, T., et al. (2020) · European Journal of Neurology · doi:10.1111/ene.14333 · Cited by 51
“Using PROBAST [33], the multivariable logistic regression analysis was adjusted for all potential confounders with a statistically significant association at P 0.1 in the univariate analysis or those confounders that may be related to END in the light of clinical experience and previous research.”
MethodInfluentialKouli, O., Murray, V., Bhatia, S., Cambridge, W. A., Kawka, M., Shafi, S., Knight, S., Kamarajah, S. K., McLean, K. A., Glasbey, J. C., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00069-3 · Cited by 56
Montazeri, M., Montazeri, M., Bahaadinbeigy, K., Montazeri, M., & Afraz, A. (2022) · Health Science Reports · doi:10.1002/hsr2.962 · Cited by 55
“We critically reviewed studies for ROB using PROBAST.(36) Our analysis revealed that except for six, all studies had some bias due to a low number of participants, lack of external validation, and failure to meet the study's goal (Table 1).”
MethodInfluentialMultimodal Deep Learning-Based Prognostication in Glioma Patients: A Systematic Review
Alleman, K., Knecht, E. M., Huang, J., Zhang, L., Lam, S. K., & DeCuypere, M. G. (2023) · Cancers · doi:10.3390/cancers15020545 · Cited by 55
“One study [13] also used treatment type, including chemoradiation, chemotherapy, targeted molecular therapy, and immunotherapy.”
MethodMachine learning models in predicting graft survival in kidney transplantation: meta-analysis
Ravindhran, B., Chandak, P., Schafer, N., Kundalia, K., Hwang, W., Antoniadis, S., Haroon, U., & Zakri, R. H. (2023) · BJS Open · doi:10.1093/bjsopen/zrad011 · Cited by 54
Romijn, M., Dhiman, P., Finken, M. J. J., van Kaam, A. H., Katz, T. A., Rotteveel, J., Schuit, E., Collins, P. G. S., Onland, W., & Torchin, H. (2023) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2023.01.024 · Cited by 53
“8 We noted that all studies had a high RoB, and most studies had a high RoB in the analysis domain.”
BackgroundDeep learning for pneumothorax diagnosis: a systematic review and meta-analysis
Sugibayashi, T., Walston, S. L., Matsumoto, T., Mitsuyama, Y., Miki, Y., & Ueda, D. (2023) · European Respiratory Review · doi:10.1183/16000617.0259-2022 · Cited by 55
Fu, H., Hou, D., Xu, R., You, Q., Li, H., Yang, Q., Wang, H., Gao, J., & Bai, D. (2023) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2023.104623 · Cited by 54
Wearable Artificial Intelligence for Detecting Anxiety: Systematic Review and Meta-Analysis
Abd-Alrazaq, A. A., AlSaad, R., Harfouche, M., Aziz, S., Ahmed, A., Damseh, R., & Sheikh, J. I. (2023) · Journal of Medical Internet Research · doi:10.2196/48754 · Cited by 65
“…of the included studies, we adapted a well-known tool (Quality Assessment of Diagnostic Accuracy Studies–Revised; QUADAS-2) [25] for our analysis by replacing some irrelevant criteria with more relevant criteria from another applicable tool (the Prediction Model Risk of Bias Assessment Tool) [26].”
Carrick, R. T., Park, J. G., McGinnes, H. L., Lundquist, C. M., Brown, K., Janes, W., Wessler, B. S., & Kent, D. M. (2020) · Journal of the American Heart Association · doi:10.1161/jaha.119.017625 · Cited by 50
Perry, B. I., Osimo, E. F., Upthegrove, R., Mallikarjun, P. K., Yorke, J., Štochl, J., Pérez, J. M., Zammit, S., Howes, O. D., Jones, P. B., et al. (2021) · The Lancet Psychiatry · doi:10.1016/s2215-0366(21)00114-0 · Cited by 56
“We excluded participants who had the outcome at baseline, as recommended; 27 however, since predictors were assessed within a short timeframe after EIS enrolment, some metabolically sensitive individuals might have been excluded from our analysis.”
Khatter, A., Naughton, M., Dambha‐Miller, H., & Redmond, P. (2021) · Learned Publishing · doi:10.1002/leap.1403 · Cited by 54
Machine learning applied to electronic health record data in home healthcare: A scoping review
Hobensack, M., Song, J., Scharp, D., Bowles, K. H., & Topaz, M. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104978 · Cited by 54
Risk prediction of cardiovascular disease in the Asia-Pacific region: the SCORE2 Asia-Pacific model
SCORE2 Asia-Pacific writing group, Abdullah, N., Jalal, M. I. A., Barr, E. L., Chamnan, P., Chong, C. L., Cuenza, L. R., Gao, P., Graham, I. D., Hilal, S., et al. (2024) · European Heart Journal · doi:10.1093/eurheartj/ehae609 · Cited by 64
“We adopted analytical approaches and reporting standards recommended by the PROBAST guidelines 26 and TRIPOD.”
Sourlos, N., Vliegenthart, R., Santinha, J., Klontzas, M. E., Cuocolo, R., Huisman, M., & van Ooijen, P. M. A. (2024) · Insights into Imaging · doi:10.1186/s13244-024-01833-2 · Cited by 56
“Efforts should be made to ensure that the risk of bias is low by considering possible factors of bias during dataset creation [30].”
BackgroundMethodMcAleenan, A., Kelly, C., Spiga, F., Kernohan, A., Cheng, H., Dawson, S., Schmidt, L., Robinson, T., Brandner, S., Faulkner, C., et al. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013316.pub2 · Cited by 57
Systematic review of prediction models in relapsing remitting multiple sclerosis
Brown, F. S., Glasmacher, S. A., Kearns, P. K. A., MacDougall, N., Hunt, D. P. J., Connick, P., & Chandran, S. (2020) · PLoS ONE · doi:10.1371/journal.pone.0233575 · Cited by 54
“Blinding outcome assessors to predictor information is especially for preventing bias when assessments are subjective and require interpretation, as is the case with many of the clinical outcomes employed here [21, 53].”
InfluentialKreuzberger, N., Damen, J. A., Trivella, M., Estcourt, L. J., Aldin, A., Umlauff, L., Vazquez-Montes, M., Wolff, R. F., Moons, K. G. M., Monsef, I., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012022.pub2 · Cited by 66
Self-citationMedina‐Lara, A., Grigore, B., Lewis, R. A., Peters, J., Price, S., Landa, P., Robinson, S., Neal, R. D., Hamilton, W. T., & Spencer, A. E. (2020) · Health Technology Assessment · doi:10.3310/hta24660 · Cited by 56
Systematic review of predictive mathematical models of COVID-19 epidemic
Shankar, S., Mohakuda, S. S., Kumar, A., Nazneen, P. S., Yadav, A. K., Chatterjee, K. D., & Chatterjee, K. (2021) · Medical Journal Armed Forces India · doi:10.1016/j.mjafi.2021.05.005 · Cited by 49
“Risk of bias in mathematical model is difficult to assess including with the use of PROBAST checklist for predictive models in medical science, as there is no selection and measurement bias in mathematical models, only the approach and type of models may differ.(7) Quality assessment of the study articles was carried…”
A Review of Research-Supported Group Treatments for Drug Use Disorders
López, G., Orchowski, L. M., Reddy, M. K., Nargiso, J., & Johnson, J. E. (2021) · Substance Abuse Treatment, Prevention, and Policy · doi:10.1186/s13011-021-00371-0 · Cited by 48
“Additionally, it is important to note that there is potential for risk of bias in the studies included across four domains: participants, predictors, outcome, and analysis [111].”
BackgroundFeature Selection based Artificial Intelligence Techniques for the Prediction of COVID like Diseases
Dutta, P., Paul, S., Obaid, A. J., Pal, S., & Mukhopadhyay, K. (2021) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1963/1/012167 · Cited by 55
Mullin, D. S., Cockburn, A., Welstead, M., Luciano, M., Russ, T. C., & Terrera, G. M. (2022) · Alzheimer s & Dementia · doi:10.1002/alz.12547 · Cited by 52
van der Heijden, A. A. W. A., Nijpels, G., Badloe, F. M. S., Lovejoy, H. L., Peelen, L. M., Feenstra, T., Moons, K. G. M., Slieker, R. C., Herings, R. M. C., Elders, P. J. M., et al. (2020) · Diabetologia · doi:10.1007/s00125-020-05134-3 · Cited by 56
“CHARMS Checklist for critical appraisal and data extraction for systemic reviews of prediction modelling studies DCS Diabetes Care System O/E Ratio of the observed and expected retinopathy events PROBAST Prediction model risk of bias assessment tool TRIPOD Transparent reporting of a multivariable prediction model for…”
MethodSelf-citationCohen, J., Wright-Berryman, J. L., Rohlfs, L., Wright, D., Campbell, M., Gingrich, D., Santel, D., & Pestian, J. P. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17218187 · Cited by 57
“After an ML model is trained, different evaluation strategies and metrics are used to evaluate performance on data that was not used to train the model [34,35].”
MethodRisk models for recurrence and survival after kidney cancer: a systematic review
Usher‐Smith, J. A., Li, L., Roberts, L. E., Harrison, H., Rossi, S. H., Sharp, S. J., Coupland, C. A. C., Hippisley‐Cox, J., Griffin, S. J., Klatte, T., et al. (2021) · British Journal of Urology · doi:10.1111/bju.15673 · Cited by 51
Manlhiot, C., den Eynde, J. V., Kutty, S., & Ross, H. J. (2021) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2021.11.009 · Cited by 55
Oprescu, A. M., Amarante, G. M., García‐Díaz, L., Rey, V. E., Chimenea, Á., Martinez-Martinez, R. L., & Ternero, M. D. C. R. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.011 · Cited by 55
Kahwati, L. C., Kistler, C. E., Booth, G., Sathe, N. A., Gordon, R. D., Okah, E. D., Wines, R., & Viswanathan, M. (2025) · JAMA · doi:10.1001/jama.2024.21653 · Cited by 58
Sharma, G., Yaffe, M. J., Ghadiri, P., Gandhi, R., Pinkham, L., Gore, G., & Abbasgholizadeh-Rahimi, S. (2025) · JMIR Mental Health · doi:10.2196/70438 · Cited by 38
“This tool evaluates the ROB and usability of diagnostic and prognostic predictions from studies that used AI models [40].”
InfluentialBridge, J., Meng, Y., Zhao, Y., Du, Y., Zhao, M., Sun, R., & Zheng, Y. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3012383 · Cited by 47
“In this work, we have strived to follow best practices for prediction modes, as outlined by PROBAST [39] and TRIPOD [60], and would encourage new prediction models in AI to follow PROBAST and TRIPOD guidelines carefully to increase the robustness of developed models.”
BackgroundMethodResultInfluentialStewart, J. E., Lü, J., Goudie, A., Bennamoun, M., Sprivulis, P., Sanfilippo, F. M., & Dwivedi, G. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252612 · Cited by 53
“Including a variable used in the definition of MI as an input in a ML model to predict MI is problematic and will likely lead to optimistic estimates of model performance [30].”
MethodInfluentialArtificial intelligence in dentistry
Meghil, M. M., Rajpurohit, P., Awad, M. E., McKee, J., Shahoumi, L. A., & Ghaly, M. (2022) · Dentistry Review · doi:10.1016/j.dentre.2021.100009 · Cited by 63
Machine learning methods for the study of cybersickness: a systematic review
Yang, A. H. X., Kasabov, N. K., & Çakmak, Y. Ö. (2022) · Brain Informatics · doi:10.1186/s40708-022-00172-6 · Cited by 50
“The ROB covered four domains from the PROBAST recommendations each containing their own signalling question items to judge risk of bias [41].”
Burghart, M., de Ruiter, C., Hynes, S. E., Krishnan, N., Levtova, Y., & Uyar, A. (2022) · Psychological Assessment · doi:10.1037/pas0001184 · Cited by 50
Altham, C., Zhang, H., & Pereira, E. (2024) · PLoS ONE · doi:10.1371/journal.pone.0303644 · Cited by 55
“The risk of bias of all included studies was assessed based on the Prediction model Risk Of Bias ASsessment Tool (PROBAST) [112].”
Forensic mental health: envisioning a more empirical future
Tully, J., Hafferty, J. D., Whiting, D., Dean, K., & Fazel, S. (2024) · The Lancet Psychiatry · doi:10.1016/s2215-0366(24)00164-0 · Cited by 51
Liu, T., Krentz, A. J., Huo, Z., & Ćurčin, V. (2025) · Reviews in Cardiovascular Medicine · doi:10.31083/rcm37443 · Cited by 55
“For the risk of bias assessment, three used QUADAS (Quality Assessment of Diagnostic Accuracy Studies) [40], two employed PROBAST (Prediction Model Risk of Bias Assessment Tool) [41], and two utilized self-designed methods derived from well-recognized techniques [20,24].”
Davenport, C. F., Rai, N., Sharma, P., Deeks, J. J., Berhane, S., Mallett, S., Saha, P., Champaneria, R., Bayliss, S. E., Snell, K. I. E., et al. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011964.pub2 · Cited by 63
Self-citationWheeler, S. M., Bryant, A., Bonney, E. A., & Howell, E. A. (2021) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2021.11.023 · Cited by 45
“80,81 These initiatives are an important step forward toward a more antiracist approach to medicine.”
Prediction Models for Intravenous Immunoglobulin Resistance in Kawasaki Disease: A Meta-analysis
Kuniyoshi, Y., Tsujimoto, Y., Banno, M., Taito, S., Ariie, T., Takahashi, N., Tokutake, H., & Takada, T. (2023) · PEDIATRICS · doi:10.1542/peds.2022-059175 · Cited by 46
Metadata
- Authors
- Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group†
- Affiliations
- Kleijnen Systematic Reviews (United Kingdom) (GB)
- Utrecht University (NL)
- Keele University (GB)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom (G.S.C.)
- Maastricht University (NL)
- Institute of Applied Health Research, National Institute for Health Research Birmingham Biomedical Research Centre, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom (S.M.)
- ORCID
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2019-01-01
- Volume
- 170
- Issue
- 1
- Pages
- 51-58
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W2907554860
- Cited by
- 2,984
- References
- 42
- Citations by year
- 2026: 921 · 2025: 561 · 2024: 416 · 2023: 343 · 2022: 303 · 2021: 250 · 2020: 140 · 2019: 45
- FWCI
- 83.39
- Type
- journal article
- Language
- en
- Open access
- Licence
- other-oa
- www.acpjournals.org/journal/aim/text-and-data-mining (tdm)
- Locations
- Annals of Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- Bristol Research (University of Bristol) (acceptedVersion, other-oa)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- FreiDok plus (Universitätsbibliothek Freiburg) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- Radboud Repository (Radboud University) (submittedVersion)
- Explore Bristol Research (submittedVersion, other-oa)
- Apollo (University of Cambridge) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Apollo
- Funders
- National Institute for Health Research Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (ZONMW 918.10.615, ZONMW 912.08.004)
- National Institute for Health Research School for Primary Care Research (390)
- National Institute for Health Research Biomedical Research Centre, Oxford
- National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) West at University Hospitals Bristol NHS Foundation Trust
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- PROBAST, risk of bias assessment, diagnostic model, prognostic model, applicability, critical appraisal, reporting guidelines, Delphi consensus, systematic reviews, personalized prediction
- MeSH terms
- Systematic Reviews as Topic, Decision Support Techniques, Delphi Technique, Diagnosis, Humans, Prognosis, Research Design, Models, Statistical, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.