The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Open accessAbstract
Clinical prediction models combine multiple predictors to estimate risk for the presence of a particular condition (diagnostic models) or the occurrence of a certain event in the future (prognostic models). PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool for assessing the risk of bias (ROB) and applicability of diagnostic and prognostic prediction model studies, was developed by a steering group that considered existing ROB tools and reporting guidelines. The tool was informed by a Delphi procedure involving 38 experts and was refined through piloting. PROBAST is organized into the following 4 domains: participants, predictors, outcome, and analysis. These domains contain a total of 20 signaling questions to facilitate structured judgment of ROB, which was defined to occur when shortcomings in study design, conduct, or analysis lead to systematically distorted estimates of model predictive performance. PROBAST enables a focused and transparent approach to assessing the ROB and applicability of studies that develop, validate, or update prediction models for individualized predictions. Although PROBAST was designed for systematic reviews, it can be used more generally in critical appraisal of prediction model studies. Potential users include organizations supporting decision making, researchers and clinicians who are interested in evidence-based medicine or involved in guideline development, journal editors, and manuscript reviewers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329 · cited 3×
“A domain-based structure was adopted similar to that used in other risk of bias tools such as the revised Cochrane Risk of Bias tool,(29) QUADAS-2,(36) ROBINSI(30) and ROBIS.”
BackgroundMethodSelf-citationQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199 · cited 3×
“A domain-based structure was adopted similar to that used in other risk of bias tools such as the revised Cochrane Risk of Bias tool,(29) QUADAS-2,(36) ROBINSI(30) and ROBIS.”
InfluentialSelf-citationCochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,178
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · cited 11×
“A multivariable prediction model is defined as any combination or equation of two or more predictors for estimating the probability or risk for an individual.(7, 8, 33-35)”
InfluentialSelf-citation2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.005 · Cited by 4,945
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Circulation · doi:10.1161/01.cir.0000437741.48606.98 · Cited by 3,697
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Self-citationAssessing Bias in Studies of Prognostic Factors
Hayden, J. A., van der Windt, D. A., Cartwright, J. L., Côté, P., & Bombardier, C. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-4-201302190-00009 · Cited by 3,287 · cited 1×
“(25, 26) The QUIPS tool has been developed to assess the risk of bias in predictor finding (prognostic factor) studies.(28) The methodological quality of studies investigating the impact of a prediction model using a comparative randomised design can be assessed using the revised Cochrane risk of bias tool (ROB 2.”
MethodCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Self-citationSystematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
Cited by
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
Self-citationCalculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917
Self-citationSCORE2 working group and ESC Cardiovascular risk collaboration, Hageman, S., Pennells, L., Ojeda, F., Kaptoge, S., Kuulasmaa, K., de Vries, T., Xu, Z., Kee, F., Chung, R., et al. (2021) · European Heart Journal · doi:10.1093/eurheartj/ehab309 · Cited by 1,810
Self-citationMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Self-citationMa, L.-L., Wang, Y.-Y., Yang, Z.-H., Huang, D., Weng, H., & Zeng, X.-T. (2020) · Military Medical Research · doi:10.1186/s40779-020-00238-8 · Cited by 1,648
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152
Moons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-082505 · Cited by 1,069
Self-citationRoberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00307-0 · Cited by 1,044
Roberts, M. T., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Avilés-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2020) · Apollo (University of Cambridge) · doi:10.17863/cam.64000 · Cited by 895
External validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887
“Bias in prognostic models can be assessed using the Prediction model Risk Of Bias ASsessment Tool tool [31].”
MethodCollins, G. S., Dhiman, P., Andaur Navarro, C. L., Ma, J., Hooft, L., Reitsma, J. B., Logullo, P., Beam, A. L., Peng, L., Van Calster, B., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-048008 · Cited by 852
Self-citationMachine learning for medical imaging: methodological failures and recommendations for the future
Varoquaux, G., & Cheplygina, V. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00592-y · Cited by 656
“The medical-research literature has several reporting guidelines for prediction studies 67,94,95 .”
Addressing bias in artificial intelligence in health care
Parikh, R. B., Teeple, S., & Navathe, A. S. (2019) · JAMA · doi:10.1001/jama.2019.18058 · Cited by 764
Muka, T., Glisic, M., Milic, J., Verhoog, S., Bohlius, J., Bramer, W., Chowdhury, R., & Franco, O. H. (2020) · European Journal of Epidemiology · doi:10.1007/s10654-019-00576-5 · Cited by 660
SCORE2-OP working group and ESC Cardiovascular risk collaboration, de Vries, T. I., Cooney, M. T., Selmer, R. M., Hageman, S. H. J., Pennells, L. A., Wood, A., Kaptoge, S., Xu, Z., Westerink, J., et al. (2021) · European Heart Journal · doi:10.1093/eurheartj/ehab312 · Cited by 622
Heller, N., Isensee, F., Maier-Hein, K. H., Hou, X., Xie, C., Li, F., Nan, Y., Mu, G., Lin, Z., Han, M., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101821 · Cited by 579
Bias in artificial intelligence algorithms and recommendations for mitigation
Nazer, L. H., Zatarah, R., Waldrip, S., Ke, J. X. C., Moukheiber, M., Khanna, A. K., Hicklen, R. S., Moukheiber, L., Moukheiber, D., Ma, H., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000278 · Cited by 621
“A number of comprehensive frameworks and checklists have been developed, such as the Translational Evaluation of Healthcare AI (TEHAI) [46], the DECIDE-AI [47], the Consolidated Standards of Reporting Trials-Artificial Intelligence (CONSORT-AI) [48], the prediction model risk of bias assessment tool (PROBAST) [49], an…”
MethodFairness of artificial intelligence in healthcare: review and recommendations
Ueda, D., Kakinuma, T., Fujita, S., Kamagata, K., Fushimi, Y., Ito, R., Matsui, Y., Nozaki, T., Nakaura, T., Fujima, N., et al. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01474-3 · Cited by 607
“This common scenario means that estimations of AI performance based on internal test data may overestimate its real‑world performance on external data [73–75].”
BackgroundBias in medical AI: Implications for clinical decision-making
Cross, J. L., Choma, M. A., & Onofrey, J. A. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000651 · Cited by 607
“…Risk of Bias Assessment (PROBAST) tool have been developed to assess the risk of imbalanced data biases, which allows both developers and downstream users (e.g., physicians, hospitals) to better understand the suitability and limitations of AI-based systems in specific clinical settings [54–56].”
MethodEvaluation of clinical prediction models (part 2): how to undertake an external validation study
Riley, R. D., Archer, L., Snell, K. I. E., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L. J., & Collins, G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074820 · Cited by 401
“…predicted values for each Box 1: Signalling questions within the first three domains of PROBAST (Prediction model Risk Of Bias ASsessment Tool) 19 20 that are important to consider when ensuring a dataset for external validation is fit for purpose • Was the outcome defined and determined in a…”
BackgroundInfluentialSelf-citationDeveloping clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544
“10-14 PROBAST (prediction model risk-of-bias assessment tool) provides a structured way to assess the risk of bias in a prediction modelling study.”
InfluentialMikkelsen, M. E., Still, M., Anderson, B. J., Bienvenu, O. J., Brodsky, M. B., Brummel, N., Butcher, B., Clay, A. S., Felt, H., Ferrante, L. E., et al. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004586 · Cited by 458
Foroutan, F., Guyatt, G., Zuk, V., Vandvik, P. O., Alba, A. C., Mustafa, R., Vernooij, R., Arevalo-Rodriguez, I., Munn, Z., Roshanov, P., et al. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.12.023 · Cited by 465
Self-citationAndaur Navarro, C. L., Damen, J. A. A., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., Moons, K. G. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2281 · Cited by 415
Self-citationGerry, S., Bonnici, T., Birks, J., Kirtley, S., Virdee, P. S., Watkinson, P. J., & Collins, G. S. (2020) · BMJ · doi:10.1136/bmj.m1501 · Cited by 383
“23 24 We assessed risk of bias by using PROBAST 30 and found that most of the included studies were at risk of bias owing to participant selection, outcome definitions, and statistical analysis.”
MethodInfluentialArtificial Intelligence in Fracture Detection: A Systematic Review and Meta-Analysis
Kuo, R. Y. L., Harrison, C., Curran, T.-A., Jones, B., Freethy, A., Cussons, D., Stewart, M., Collins, G. S., & Furniss, D. (2022) · Radiology · doi:10.1148/radiol.211785 · Cited by 365
Guo, C., Ashrafian, H., Ghafur, S., Fontana, G., Gardner, C., & Prime, M. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00314-2 · Cited by 316
QUADAS-C: A Tool for Assessing Risk of Bias in Comparative Diagnostic Accuracy Studies
Yang, B., Mallett, S., Takwoingi, Y., Davenport, C. F., Hyde, C. J., Whiting, P. F., Deeks, J. J., Leeflang, M. M. G., & the QUADAS-C Group† (2021) · Annals of Internal Medicine · doi:10.7326/m21-2234 · Cited by 324
“For example, PROBAST (Prediction model Risk Of Bias ASsessment Tool) (24) provides more specific signaling questions for multivariable models that users could consider when tailoring.”
BackgroundSelf-citationBias recognition and mitigation strategies in artificial intelligence healthcare applications
Hasanzadeh, F., Josephson, C. B., Waters, G., Adedinsewo, D., Azizi, Z., & White, J. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01503-7 · Cited by 320
“A similar study, performed by Chen, et al., using the PROBAST (Prediction model Risk Of Bias ASsessment Tool) framework 17 (For further details, refer to Supplementary Table 1), examined 555 published neuroimaging-based AI models for psychiatric diagnosis, identifying only 86 studies (15.5”
Influentialde Souza Barbosa, A., da Silva, M. C. B. C., da Silva, L. B., Morioka, S. N., & de Souza, V. F. (2023) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-01919-0 · Cited by 203
Ethical use of Artificial Intelligence in Health Professions Education: AMEE Guide No. 158
Masters, K. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2186203 · Cited by 302
“Tools are also being developed to check for data and algorithm bias (e.g. PROBAST: Wolff et al. 2019), although more directed tools are required.”
BackgroundThe need to separate the wheat from the chaff in medical informatics
Cabitza, F., & Campagner, A. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104510 · Cited by 276
COVID-19 open source data sets: a comprehensive survey
Shuja, J., Alanazi, E., Alasmary, W., & Alashaikh, A. (2021) · Applied Intelligence · doi:10.1007/s10489-020-01862-6 · Cited by 274
“The critical analysis utilized PROBAST, a tool for risk and bias assessment in prediction models [93].”
MethodIlluminating the dark spaces of healthcare with ambient intelligence
Haque, A., Milstein, A., & Fei-Fei, L. (2020) · Nature · doi:10.1038/s41586-020-2669-y · Cited by 274
“Nonetheless, progress is being made to mitigate bias, such as the PROBAST tool 158 .”
BackgroundMitigating bias in machine learning for medicine
Vokinger, K. N., Feuerriegel, S., & Kesselheim, A. S. (2021) · Communications Medicine · doi:10.1038/s43856-021-00028-w · Cited by 269
Use of machine learning to analyse routinely collected intensive care unit data: a systematic review
Shillan, D., Sterne, J. A. C., Champneys, A., & Gibbison, B. (2019) · Critical Care · doi:10.1186/s13054-019-2564-9 · Cited by 251
Damen, J. A., Pajouheshnia, R., Heus, P., Moons, K. G. M., Reitsma, J. B., Scholten, R. J. P. M., Hooft, L., & Debray, T. P. A. (2019) · BMC Medicine · doi:10.1186/s12916-019-1340-7 · Cited by 250
“Risk of bias was assessed based on a combination of the CHARMS checklist [23] and a preliminary version of the Cochrane Prediction study Risk Of Bias Assessment Tool (PROBAST) [27, 28] (Additional file 5).”
MethodSelf-citationArtificial Intelligence and Liability in Medicine: Balancing Safety and Innovation
MALIHA, G., GERKE, S., COHEN, I. G., & PARIKH, R. B. (2021) · The Milbank Quarterly · doi:10.1111/1468-0009.12504 · Cited by 211
Buchlak, Q. D., Esmaili, N., Leveque, J.-C., Farrokhi, F., Bennett, C., Piccardi, M., & Sethi, R. K. (2020) · Neurosurgical Review · doi:10.1007/s10143-019-01163-8 · Cited by 227
“Risk of bias was assessed using the Prediction model Risk of Bias Assessment Tool (PROBAST) [118].”
MethodSalazar de Pablo, G., Studerus, E., Vaquerizo-Serrano, J., Irving, J., Catalan, A., Oliver, D., Baldwin, H., Danese, A., Fazel, S., Steyerberg, E. W., et al. (2021) · Schizophrenia Bulletin · doi:10.1093/schbul/sbaa120 · Cited by 219
“16, 25–52 We further conducted metaanalytical regressions to estimate the association between accuracy and (1) the type of validation (internal vs external); (2) the type of accuracy measure (area under the curve [AUC] vs C-statistics vs accuracy, with the latter category including accuracy measures other than AUC or…”
Evaluation of ChatGPT-generated medical responses: A systematic review and meta-analysis
Wei, Q., Yao, Z., Cui, Y., Wei, B., Jin, Z., & Xu, X. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104620 · Cited by 201
Gupta, R. K., Harrison, E. M., Ho, A., Docherty, A. B., Knight, S. R., van Smeden, M., Abubakar, I., Lipman, M., Quartagno, M., Pius, R., et al. (2021) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(20)30559-2 · Cited by 210
Meehan, A. J., Lewis, S. J., Fazel, S., Fusar-Poli, P., Steyerberg, E. W., Stahl, D., & Danese, A. (2022) · Molecular Psychiatry · doi:10.1038/s41380-022-01528-4 · Cited by 219
“Quality assessment For each individual model development and external validation analysis, the Prediction model Risk Of Bias ASsessment Tool (PROBAST; see Appendix 5 of Supplementary Material) was used to assess risk of bias (ROB), or the potential for inflated model performance estimates due to shortcomings in design…”
BackgroundMethodInfluentialSnell, K. I. E., Levis, B., Damen, J. A. A., Dhiman, P., Debray, T. P. A., Hooft, L., Reitsma, J. B., Moons, K. G. M., Collins, G. S., & Riley, R. D. (2023) · BMJ · doi:10.1136/bmj-2022-073538 · Cited by 190
“…studies, 22 PROBAST (prediction model risk-of-bias assessment tool) for evaluating the quality and applicability of prediction model studies, 23 24 statistical methods for obtaining estimates and confidence intervals of model performance measures from each study, 18 25 meta-analysis methods…”
MethodSelf-citationKim, M., Chen, C., Wang, P., Mulvey, J. J., Yang, Y., Wun, C., Antman-Passig, M., Luo, H.-B., Cho, S., Long-Roche, K., et al. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00860-y · Cited by 214
Stability of clinical prediction models developed using statistical or machine learning methods
Riley, R. D., & Collins, G. S. (2023) · Biometrical Journal · doi:10.1002/bimj.202200302 · Cited by 211
Self-citationTakita, H., Kabata, D., Walston, S. L., Tatekawa, H., Saito, K., Tsujimoto, Y., Miki, Y., & Ueda, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01543-z · Cited by 203
“We used PROBAST to assess papers for bias and applicability 114 .”
Influentialten Hoorn, S., de Back, T. R., Sommeijer, D. W., & Vermeulen, L. (2022) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djab106 · Cited by 203
“The Prediction model Risk Of Bias ASsessment Tool was used to assess the risk of bias of the included full-text studies (17).”
MethodStirland, L. E., González-Saavedra, L., Mullin, D. S., Ritchie, C. W., Muniz-Terrera, G., & Russ, T. C. (2020) · BMJ · doi:10.1136/bmj.m160 · Cited by 189
Quinn, T. J., Richard, E., Teuschl, Y., Gattringer, T., Hafdi, M., O'Brien, J. T., Merriman, N., Gillebert, C., Huygelier, H., Verdelho, A., et al. (2021) · European Journal of Neurology · doi:10.1111/ene.15068 · Cited by 182
Artificial Intelligence for COVID-19: A Systematic Review
Wang, L., Zhang, Y., Wang, D., Tong, X., Liu, T., Zhang, S., Huang, J., Zhang, L., Chen, L., Fan, H., et al. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.704256 · Cited by 182
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198
Self-citationBellou, V., Belbasis, L., Konstantinidis, A. K., Tzoulaki, I., & Evangelou, E. (2019) · BMJ · doi:10.1136/bmj.l5358 · Cited by 197
Gupta, R. K., Marks, M., Samuels, T. H. A., Luintel, A., Rampling, T., Chowdhury, H., Quartagno, M., Nair, A., Lipman, M., Abubakar, I., et al. (2020) · European Respiratory Journal · doi:10.1183/13993003.03498-2020 · Cited by 199
“Critical appraisal of these models using quality assessment tools developed specifically for prediction modelling studies suggests that the candidate models are poorly reported and at high risk of bias and overestimation of their reported performance [8, 9].”
Evaluation and Mitigation of Racial Bias in Clinical Machine Learning Models: Scoping Review
Huang, J., Galal, G., Etemadi, M., & Vaidyanathan, M. (2022) · JMIR Medical Informatics · doi:10.2196/36388 · Cited by 186
“An increased focus from medical journals on bias evaluation checklists applicable to clinical ML models, such as the Prediction Model Risk of Bias Assessment Tool (PROBAST), is desirable to further emphasize vigilance regarding biased ML models [46].”
Bringing MRI to low‐ and middle‐income countries: Directions, challenges and potential solutions
Murali, S., Ding, H., Adedeji, F., Qin, C., Obungoloch, J., Asllani, I., Anazodo, U., Ntusi, N. A. B., Mammen, R., Niendorf, T., et al. (2024) · NMR in Biomedicine · doi:10.1002/nbm.4992 · Cited by 163
“186,193 194 These included a lack of infrastructure, which allows them to travel, the cost of travelling to the hospital/care facility and an inability to leave home for a prolonged time to seek healthcare.”
BackgroundMachine learning in medicine: a practical introduction to natural language processing
Harrison, C. J., & Sidey-Gibbons, C. J. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01347-1 · Cited by 168
Chen, F., Wang, L., Hong, J., Jiang, J., & Zhou, L. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae060 · Cited by 178
“Our analyses were further enhanced by consolidating established bias risk assessment tools, such as ROBINS-I, 16 ROBINS-E, 17 and PROBAST, 18 and relevant review articles, such as Mehrabi et al 19 which categorized potential biases in AI applications.”
Liu, T., Krentz, A., Lu, L., & Curcin, V. (2025) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztae080 · Cited by 177
Maleki, F., Ovens, K. L., Gupta, R., Reinhold, C., Spatz, A., & Forghani, R. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.220028 · Cited by 223
“designed PROBAST as a series of questions to facilitate systematic review and assessment of potential bias in clinical prediction models (16).”
BackgroundIdentifying predictors of ventral hernia recurrence: systematic review and meta-analysis
Parker, S. G., . Mallett, S., Quinn, L., Wood, C. P. J., Boulton, R. W., Jamshaid, S., . Erotocritou, M., . Gowda, S., . Collier, W., Plumb, A. A. O., et al. (2021) · BJS Open · doi:10.1093/bjsopen/zraa071 · Cited by 163
Self-citationRahimi, S. A., Légaré, F., Sharma, G., Archambault, P. M., Zomahoun, H. T. V., Chandavong, S., Rhéault, N., Wong, S. T., Langlois, L., Couturier, Y., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/29839 · Cited by 192
“If one or more domains were judged as having high risk of bias, the overall judgment was “high risk” [31].”
InfluentialFrom real-world electronic health record data to real-world results using artificial intelligence
Knevel, R., & Liao, K. P. (2022) · Annals of the Rheumatic Diseases · doi:10.1136/ard-2022-222626 · Cited by 193
“49 The length of the patients’ trajectories influences the chance of being captured in case-control studies.”
BackgroundKoçak, B., Ponsiglione, A., Stanzione, A., Bluethgen, C., Santinha, J., Ugga, L., Huisman, M., Klontzas, M. E., Cannella, R., & Cuocolo, R. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242854 · Cited by 168
“60 Nonetheless, the current version of PROBAST is not suitable for the evaluation of ML studies, and this is the reason that the PROBAST group has initiated the process of developing an AI-specific version of PROBAST called PROBAST-AI, which is still under development.”
Brain–phenotype models fail for individuals who defy sample stereotypes
Greene, A. S., Shen, X., Noble, S., Horien, C., Hahn, C. A., Arora, J., Tokoglu, F., Spann, M. N., Carrión, C. I., Barron, D. S., et al. (2022) · Nature · doi:10.1038/s41586-022-05118-w · Cited by 167
Defining representativeness of study samples in medical and population health research
Rudolph, J. E., Zhong, Y., Duggal, P., Mehta, S. H., & Lau, B. M. (2023) · BMJ Medicine · doi:10.1136/bmjmed-2022-000399 · Cited by 194
“16 17 All concepts centre on the idea that it is important to assess a study and its results in terms of how well they can be related to some target population.”
BackgroundAkazawa, M., & Hashimoto, K. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102164 · Cited by 160
Machine learning for genetic prediction of psychiatric disorders: a systematic review
Bracher‐Smith, M., Crawford, K. L., & Escott‐Price, V. (2020) · Molecular Psychiatry · doi:10.1038/s41380-020-0825-2 · Cited by 177
“In addition to PROBAST, information on population structure within studies was extracted (Table S6).”
BackgroundMethodInfluentialFehr, J., Citro, B., Malpani, R., Lippert, C., & Madai, V. I. (2024) · Frontiers in Digital Health · doi:10.3389/fdgth.2024.1267290 · Cited by 187
A systematic review of few-shot learning in medical imaging
Pachetti, E., & Colantonio, S. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102949 · Cited by 197
Delgado, A. H. S., Sauro, S., Lima, A. F., Loguércio, A. D., Bona, Á. D., Mazzoni, A., Collares, F. M., Staxrud, F., Ferracane, J. L., Tsoi, J. K. H., et al. (2022) · Journal of Dentistry · doi:10.1016/j.jdent.2022.104350 · Cited by 161
Frankfurter, C., Molinero, M., Vishram‐Nielsen, J. K. K., Foroutan, F., Mak, S. S., Rao, V. A., Billia, F., Orchanian‐Cheff, A., & Alba, A. C. (2020) · Circulation Heart Failure · doi:10.1161/circheartfailure.120.006994 · Cited by 158
Marcolino, M. S., Ziegelmann, P. K., Souza-Silva, M. V. R., do Nascimento, I. J. B., Oliveira, L. M., Monteiro, L. S., Sales, T. L. S., Ruschel, K. B., Martins, K. P. M. P., Etges, A. P. B. D. S., et al. (2021) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2021.01.019 · Cited by 155
Nwanosike, E. M., Conway, B. R., Merchant, H. A., & Hasan, S. S. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104679 · Cited by 162
Recent methodological advances in federated learning for healthcare
Zhang, F., Kreuter, D., Chen, Y. K., Dittmer, S., Tull, S., Shadbahr, T., Schut, M. C., Asselbergs, F. W., Kar, S., Sivapalaratnam, S., et al. (2024) · Patterns · doi:10.1016/j.patter.2024.101006 · Cited by 185
“Additionally, tools such as PROBAST 149 are recommended for assessing the biases in the data and models.”
Drukker, K., Chen, W., Gichoya, J. W., Gruszauskas, N. P., Kalpathy-Cramer, J., Koyejo, S., Myers, K. J., Sá, R. C., Sahiner, B., Whitney, H. M., et al. (2023) · Journal of Medical Imaging · doi:10.1117/1.jmi.10.6.061104 · Cited by 149
Sanmarchi, F., Fanconi, C., Golinelli, D., Gori, D., Hernandez‐Boussard, T., & Capodici, A. (2023) · Journal of Nephrology · doi:10.1007/s40620-023-01573-4 · Cited by 176
“According to the PROBAST assessment tool [18], most of the included articles showed an overall low risk of bias (n = 48; 67.”
MethodInfluentialSufriyana, H., Husnayain, A., Chen, Y.-L., Kuo, C.-Y., Singh, O., Yeh, T.-Y., Wu, Y., & Su, E. C. (2020) · JMIR Medical Informatics · doi:10.2196/16503 · Cited by 168
“Applicability was assessed using assessments that were a part of the prediction model risk of bias assessment tool (PROBAST) [17,21].”
BackgroundMethodSouaifi, M., Dhahbi, W., Jebabli, N., Ceylan, H. İ., Boujabli, M., Muntean, R. I., & Dergaa, I. (2025) · Bioengineering · doi:10.3390/bioengineering12080887 · Cited by 160
“We developed an assessment framework based on established criteria from the Prediction Model Risk of Bias Assessment Tool (PROBAST) [15] and the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis guidelines [16], adapted explicitly for AI applications in…”
Dretzke, J., Chuchu, N., Agarwal, R., Herd, C. P., Chua, W., Fabritz, L., Bayliss, S. E., Kotecha, D., Deeks, J. J., Kirchhof, P. F., et al. (2020) · EP Europace · doi:10.1093/europace/euaa041 · Cited by 160
“Risk of bias was assessed using the Prediction Study Risk Of Bias Assessment Tool (PROBAST).(30) This assesses criteria within five domains: participant selection; predictors; outcomes; sample size; and patient flow and analysis (Supplementary material online, File S2).”
Frizzell, T. O., Glashutter, M., Liu, C. C., Zeng, A., Pan, D., Hajra, S. G., D’Arcy, R. C. N., & Song, X. (2022) · Ageing Research Reviews · doi:10.1016/j.arr.2022.101614 · Cited by 165
Quinn, T. J., Richard, E., Teuschl, Y., Gattringer, T., Hafdi, M., O'Brien, J. T., Merriman, N. A., Gillebert, C. R., Huygelier, H., Verdelho, A., et al. (2021) · European Stroke Journal · doi:10.1177/23969873211042192 · Cited by 166
Machine Learning in Vascular Surgery: A Systematic Review and Critical Appraisal
Li, B., Feridooni, T., Cuen-Ojeda, C., Kishibe, T., de Mestral, C., Mamdani, M., & Al-Omran, M. A. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00552-y · Cited by 149
“PROBAST and TRIPOD are designed to assess prediction models, but have not been validated speci fi cally for ML applications.”
MethodInfluentialden Bosch, L. V., van der Schaaf, A., van der Laan, H. P., Hoebers, F. J. P., Wijers, O. B., van den Hoek, J. G. M., Moons, K. G. M., Reitsma, J. B., Steenbakkers, R. J. H. M., Schuit, E., et al. (2021) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2021.01.024 · Cited by 144
Self-citationDhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., et al. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01577-x · Cited by 140
“…of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guideline, the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) and the Prediction model Risk Of Bias ASsessment Tool (PROBAST) [39, 40, 43–45].”
MethodSelf-citationSteiner, T., Purrucker, J. C., de Sousa, D. A., Apostolaki‐Hansson, T., Beck, J., Christensen, H., Cordonnier, C., Downer, M. B., Eilertsen, H., Gartly, R., et al. (2025) · European Stroke Journal · doi:10.1177/23969873251340815 · Cited by 162
Abd-Alrazaq, A., AlSaad, R., Shuweihdi, F., Ahmed, A., Aziz, S., & Sheikh, J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00828-5 · Cited by 138
“We modi fi ed a well-known risk of bias assessment tool (Quality Assessment of Studies of Diagnostic Accuracy-Revised (QUADAS-2)) 79 by removing some irrelevant criteria and adding other criteria from another relevant tool (the Prediction model Risk Of Bias ASsessment Tool (PROBAST)) 80 .”
van de Sande, D., Van Genderen, M. E., Smit, J. M., Huiskens, J., Visser, J. J., Veen, R. E. R., van Unen, E., BA, O. H., Gommers, D., & Bommel, J. V. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100495 · Cited by 135
Pandor, A., Tonkins, M., Goodacre, S. W., Sworn, K., Clowes, M., Griffin, X. L., Holland, M., Hunt, B. J., de Wit, K., & Horner, D. E. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-045672 · Cited by 144
Porto, B. M. (2024) · BMC Emergency Medicine · doi:10.1186/s12873-024-01135-2 · Cited by 162
“…main findings include: (i) ML models exhibited a high risk of bias when evaluated using the Prediction Model Risk of Bias Assessment Tool (PROBAST) [53], (ii) NLP enhanced the performance of ML algorithms in predicting patient triage, (iii) FE and XAI approaches were underutilized in this field,…”
Deep Learning and Medical Image Analysis for COVID-19 Diagnosis and Prediction
Liu, T., Siegel, E. L., & Shen, D. G. (2022) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-110220-012203 · Cited by 160
Tornero-Costa, R., Martínez-Millana, A., Muscat, N. A., Lazëri, L., Traver, V., & Novillo-Ortiz, D. (2023) · JMIR Mental Health · doi:10.2196/42045 · Cited by 161
“Risk of bias was evaluated by combining the Cochrane tool for randomized controlled trials [11] and PROBAST (Prediction Model Risk of Bias Assessment Tool) guidelines [12].”
MethodHickman, S. E., Woitek, R., Le, E., Im, Y. R., Luxhøj, C. M., Avilés-Rivero, A. I., Baxter, G. C., Mackay, J., & Gilbert, F. J. (2021) · Radiology · doi:10.1148/radiol.2021210391 · Cited by 139
Best Practices in Supervised Machine Learning: A Tutorial for Psychologists
Pargent, F., Schoedel, R., & Stachl, C. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231162559 · Cited by 163
“There is also a growing literature with specific guidelines for clinical predictive models (Moons et al., 2015; Wolff et al., 2019), which require especially high methodological and ethical standards.”
We need to talk about nonprobability samples
Boyd, R. J., Powney, G. D., & Pescott, O. L. (2023) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2023.01.001 · Cited by 144
A systematic review of federated learning applications for biomedical data
Crowson, M. G., Moukheiber, D., Arévalo, A. R., Lam, B. D., Mantena, S., Rana, A., Goss, D., Bates, D. W., & Celi, L. A. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000033 · Cited by 129
“To assess the risk-of-bias of the included studies, we utilized the Prediction model Risk Of Bias ASsessment Tool (PROBAST).[8] PROBAST was developed as a tool for systematic reviews to assess the risk-of-bias and applicability of studies describing diagnostic and prognostic prediction models.”
MethodFernandez-Felix, B. M., López‐Alcalde, J., Figuls, M. R. I., Muriel, A., & Zamora, J. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-01849-0 · Cited by 127
“The PROBAST tool (Prediction model Risk Of Bias Assessment Tool) is a checklist for assessing the risk of bias and the applicability of prognostic model studies [6, 7].”
MethodMikó, A., Vigh, É., Mátrai, P., Soós, A., Garami, A., Balaskó, M., Czakó, L., Mosdósi, B., Sarlós, P., Erőss, B. M., et al. (2019) · Frontiers in Physiology · doi:10.3389/fphys.2019.01002 · Cited by 126
“The Predictionmodel Risk Of Bias ASsessment Tool (PROBAST) (Wolff et al., 2019) was used to assess the risk of bias and applicability of primary studies in accordance with the recommendation of the Cochrane Collaboration.”
Segar, M. W., Jaeger, B. C., Patel, K. V., Nambi, V., Ndumele, C. E., Correa, A., Butler, J., Chandra, A., Ayers, C. R., Rao, S. G., et al. (2021) · Circulation · doi:10.1161/circulationaha.120.053134 · Cited by 140
Metadata
- Authors
- Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group†
- Affiliations
- Kleijnen Systematic Reviews (United Kingdom) (GB)
- Utrecht University (NL)
- Keele University (GB)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom (G.S.C.)
- Maastricht University (NL)
- Institute of Applied Health Research, National Institute for Health Research Birmingham Biomedical Research Centre, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom (S.M.)
- ORCID
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2019-01-01
- Volume
- 170
- Issue
- 1
- Pages
- 51-58
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W2907554860
- Cited by
- 2,984
- References
- 42
- Citations by year
- 2026: 921 · 2025: 561 · 2024: 416 · 2023: 343 · 2022: 303 · 2021: 250 · 2020: 140 · 2019: 45
- FWCI
- 83.39
- Type
- journal article
- Language
- en
- Open access
- Licence
- other-oa
- www.acpjournals.org/journal/aim/text-and-data-mining (tdm)
- Locations
- Annals of Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- Bristol Research (University of Bristol) (acceptedVersion, other-oa)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- FreiDok plus (Universitätsbibliothek Freiburg) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- Radboud Repository (Radboud University) (submittedVersion)
- Explore Bristol Research (submittedVersion, other-oa)
- Apollo (University of Cambridge) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Apollo
- Funders
- National Institute for Health Research Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (ZONMW 918.10.615, ZONMW 912.08.004)
- National Institute for Health Research School for Primary Care Research (390)
- National Institute for Health Research Biomedical Research Centre, Oxford
- National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) West at University Hospitals Bristol NHS Foundation Trust
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- PROBAST, risk of bias assessment, diagnostic model, prognostic model, applicability, critical appraisal, reporting guidelines, Delphi consensus, systematic reviews, personalized prediction
- MeSH terms
- Systematic Reviews as Topic, Decision Support Techniques, Delphi Technique, Diagnosis, Humans, Prognosis, Research Design, Models, Statistical, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.