PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Clinical prediction models combine multiple predictors to estimate risk for the presence of a particular condition (diagnostic models) or the occurrence of a certain event in the future (prognostic models). PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool for assessing the risk of bias (ROB) and applicability of diagnostic and prognostic prediction model studies, was developed by a steering group that considered existing ROB tools and reporting guidelines. The tool was informed by a Delphi procedure involving 38 experts and was refined through piloting. PROBAST is organized into the following 4 domains: participants, predictors, outcome, and analysis. These domains contain a total of 20 signaling questions to facilitate structured judgment of ROB, which was defined to occur when shortcomings in study design, conduct, or analysis lead to systematically distorted estimates of model predictive performance. PROBAST enables a focused and transparent approach to assessing the ROB and applicability of studies that develop, validate, or update prediction models for individualized predictions. Although PROBAST was designed for systematic reviews, it can be used more generally in critical appraisal of prediction model studies. Potential users include organizations supporting decision making, researchers and clinicians who are interested in evidence-based medicine or involved in guideline development, journal editors, and manuscript reviewers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329 · cited 3×
“A domain-based structure was adopted similar to that used in other risk of bias tools such as the revised Cochrane Risk of Bias tool,(29) QUADAS-2,(36) ROBINSI(30) and ROBIS.”
BackgroundMethodSelf-citationQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199 · cited 3×
“A domain-based structure was adopted similar to that used in other risk of bias tools such as the revised Cochrane Risk of Bias tool,(29) QUADAS-2,(36) ROBINSI(30) and ROBIS.”
InfluentialSelf-citationCochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,178
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · cited 11×
“A multivariable prediction model is defined as any combination or equation of two or more predictors for estimating the probability or risk for an individual.(7, 8, 33-35)”
InfluentialSelf-citation2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.005 · Cited by 4,945
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Circulation · doi:10.1161/01.cir.0000437741.48606.98 · Cited by 3,697
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Self-citationAssessing Bias in Studies of Prognostic Factors
Hayden, J. A., van der Windt, D. A., Cartwright, J. L., Côté, P., & Bombardier, C. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-4-201302190-00009 · Cited by 3,287 · cited 1×
“(25, 26) The QUIPS tool has been developed to assess the risk of bias in predictor finding (prognostic factor) studies.(28) The methodological quality of studies investigating the impact of a prediction model using a comparative randomised design can be assessed using the revised Cochrane risk of bias tool (ROB 2.”
MethodCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Self-citationSystematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
Cited by
Items 101–200 of 1,010
Cai, Y., Cai, Y.-Q., Tang, L.-Y., Wang, Y.-H., Gong, M., Jing, T.-C., Li, H.-J., Li-Ling, J., Hu, W., Yin, Z., et al. (2024) · BMC Medicine · doi:10.1186/s12916-024-03273-7 · Cited by 122
“The risk of bias was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST) [30], and the extraction form included four domains: participants, predictors, outcomes, and statistical analysis.”
BackgroundMethodAndaur Navarro, C. L., Damen, J. A. A., van Smeden, M., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., et al. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.11.015 · Cited by 121
Self-citationTargeted validation: validating clinical prediction models in their intended population and setting
Sperrin, M., Riley, R. D., Collins, G. S., & Martin, G. P. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00136-8 · Cited by 108
“It is also emphasised in the PROBAST risk of bias tool for systematic reviews and meta analyses of CPMs, where the ‘applicability’ domain checks whether included studies consider the same setting and population as the review question [17], emphasised in a recent scoping review of guidance for prediction models using A…”
MethodSelf-citationNeoadjuvant Chemotherapy for Newly Diagnosed, Advanced Ovarian Cancer: ASCO Guideline Update
Gaillard, S. L., Lacchetti, C., Armstrong, D. K., Cliby, W. A., Edelson, M. I., García, A. A., Ghebre, R. G., Gressel, G. M., Lesnock, J. L., Meyer, L. A., et al. (2025) · Journal of Clinical Oncology · doi:10.1200/jco-24-02589 · Cited by 143
Reproducibility in machine‐learning‐based research: Overview, barriers, and drivers
Semmelrock, H., Ross‐Hellauer, T., Kopeinik, S., Theiler, D., Haberl, A., Thalmann, S., & Kowald, D. (2025) · AI Magazine · doi:10.1002/aaai.70002 · Cited by 118
How to conduct a systematic review and meta-analysis of prognostic model studies
Damen, J. A., Moons, K. G. M., van Smeden, M., & Hooft, L. (2022) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2022.07.019 · Cited by 124
“Prediction model Risk Of Bias ASsessment tool [24,33], meta-analysis [30,34], the”
Self-citationAstridge, B., Li, W. W., McDERMOTT, B. M., & Longhitano, C. (2023) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2023.106055 · Cited by 126
Smith, L. A., Oakden-Rayner, L., Bird, A., Zeng, M., To, M.-S., Mukherjee, S., & Palmer, L. J. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00177-2 · Cited by 116
Buchlak, Q. D., Esmaili, N., Leveque, J., Bennett, C. C., Farrokhi, F. R., & Piccardi, M. (2021) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2021.04.043 · Cited by 123
Artificial intelligence-enabled decision support in nephrology
Loftus, T. J., Shickel, B., Ozrazgat‐Baslanti, T., Ren, Y., Glicksberg, B. S., Cao, J., Singh, K., Chan, L., Nadkarni, G. N., & Bihorac, A. (2022) · Nature Reviews Nephrology · doi:10.1038/s41581-022-00562-3 · Cited by 128
“Following deployment, researchers and independent oversight committees should ensure that the models provide equal clinical benefit, predictive performance and resource allocation for all patient subgroups 74,93 .”
BackgroundNußbaumer-Streit, B., Sommer, I., Hamel, C., Devane, D., Noel-Storr, A. H., Puljak, L., Trivella, M., & Gartlehner, G. (2023) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2022-112185 · Cited by 131
A Review of Artificial Intelligence and Robotics in Transformed Health Ecosystems
Denecke, K., & Baudoin, C. R. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.795957 · Cited by 95
“This can be considered “desirable bias” that counteracts the undesirable kind (89) and gets us closer to the goals of P5 medicine.”
Sufriyana, H., Wu, Y., & Su, E. C. (2020) · EBioMedicine · doi:10.1016/j.ebiom.2020.102710 · Cited by 119
“Therefore, this study applied standards for feature extraction, feature selection, model validation, and others that were designed to avoid bias and over fi tting in development of either prognostic factors or the prediction model [25,28,96,97].”
MethodDhiman, P., Ma, J., Qi, C., Bullock, G. S., Sergeant, J. C., Riley, R. D., & Collins, G. S. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-02008-1 · Cited by 125
“Future research and recommendations Sample size is one of the minimum reporting recommendations in the TRIPOD reporting guideline and signalling question for PROBAST the risk of bias tool for prediction model studies [1, 2, 34, 35].”
MethodSelf-citationHurkmans, C., Bibault, J.-E., Brock, K. K., van Elmpt, W., Feng, M., David Fuller, C., Jereczek-Fossa, B. A., Korreman, S., Landry, G., Madesta, F., et al. (2024) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2024.110345 · Cited by 110
Silva, K. D., Lee, W. K., Forbes, A., Demmer, R. T., Barton, C., & Enticott, J. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104268 · Cited by 108
Machine Learning in Dentistry: A Scoping Review
Arsiwala-Scheppach, L. T., Chaurasia, A., Müller, A., Krois, J., & Schwendicke, F. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12030937 · Cited by 110
“We note that alternatively (or even complimentary), the PROBAST tool [10] could have been used for the same assessment.”
MethodSystematic Review of Deep Learning Image Analyses for the Diagnosis and Monitoring of Skin Disease
Choy, S. P., Kim, B. J., Paolino, A., Tan, W. R., Lim, S. M. L., Seo, J., Tan, S. P., Francis, L., Tsakok, T., Simpson, M., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00914-8 · Cited by 105
Tan, K. R., Seng, J. J. B., Kwan, Y. H., Chen, Y. J., Zainudin, S. B., Loh, D. H. F., Liu, N., & Low, L. L. (2021) · Journal of Diabetes Science and Technology · doi:10.1177/19322968211056917 · Cited by 117
“34 All disagreements were resolved through discussions with a third independent reviewer (J.J.B.S.).”
BackgroundSiddiqi, T. J., Usman, M., Shahid, I., Ahmed, J., Khan, S. U., Ya’qoub, L., Rihal, C. S., & Alkhouli, M. A. (2021) · European Journal of Preventive Cardiology · doi:10.1093/eurjpc/zwab018 · Cited by 108
“The refined Prediction model Risk of Bias Assessment Tool (PROBAST) was used to assess the risk of bias in the included studies.(11) The Begg’s test was performed to evaluate publication bias in our study.”
QUAPAS: An Adaptation of the QUADAS-2 Tool to Assess Prognostic Accuracy Studies
Lee, J., Mulder, F. I., Leeflang, M., Wolff, R. F., Whiting, P. F., & Bossuyt, P. M. (2022) · Annals of Internal Medicine · doi:10.7326/m22-0276 · Cited by 111
“, censoring, competing risks, sampling of control participants] accounted for appropriately?” [11]).”
InfluentialSelf-citationCarrasco-Ribelles, L. A., Llanes-Jurado, J., Gallego-Moll, C., Cabrera‐Bean, M., Monteagudo, M., Violán, C., & Zabaleta‐del‐Olmo, E. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad168 · Cited by 121
Debray, T. P. A., Collins, G. S., Riley, R. D., Snell, K. I. E., Van Calster, B., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071018 · Cited by 86
Self-citationMurphy, R. M., Klopotowska, J. E., de Keizer, N. F., Jager, K. J., Leopold, J. H., Dongelmans, D. A., Abu-Hanna, A., & Schut, M. C. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279842 · Cited by 90
Pietraszewski, P., Terbalyan, A., Roczniok, R., Maszczyk, A., Ornowski, K., Manilewska, D., Kuliś, S., Zając, A., & Gołaś, A. (2025) · Applied Sciences · doi:10.3390/app15137254 · Cited by 119
Bryant, A., Hiu, S. K. W., Kunonga, P., Gajjar, K. B., Craig, D., Vale, L. D., Winter-Roach, B. A., Elattar, A., & Naik, R. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015048.pub2 · Cited by 149
Overview of clinical prediction models
Chen, L. (2020) · Annals of Translational Medicine · doi:10.21037/atm.2019.11.121 · Cited by 110
“Based on PROBAST (Prediction model Risk of Bias Assessment Tool) (6), dichotomization or categorization of the continuous predictors would lose information.”
MethodInfluentialKusuma, K., Larsen, M. E., Quiroz, J. C., Gillies, M. B., Burnett, A., Qian, J., & Torok, M. H. (2022) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2022.09.050 · Cited by 101
Alsaleh, M. M., Allery, F., Choi, J. W., Hama, T., McQuillin, A., Wu, H., & Thygesen, J. H. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105088 · Cited by 112
“The Prediction model study Risk of Bias Assessment Tool (PROBAST) was used to assess the Risk of Bias (ROB) and applicability of the bestperforming ML model in the selected articles [20].”
MethodArtificial intelligence in clinical care amidst COVID-19 pandemic: A systematic review
Adamidi, E. S., Mitsis, K., & Nikita, K. S. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.05.010 · Cited by 104
“In this systematic review, we used PROBAST (Prediction model Risk Of Bias ASsessment Tool) [18], to assess the risk of bias and applicability of the included studies with a focused and transparent approach.”
Evaluating risk prediction models for adults with heart failure: A systematic literature review
Tanna, G. L. D., Wirtz, H. S., Burrows, K. L., & Globe, G. A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0224135 · Cited by 104
“Introduction of bias could lead to the wrong patients being identified and treated, and ultimately costly mistakes within a healthcare system, if the model was widely used [16, 17].”
Face Mask Use in the Community for Reducing the Spread of COVID-19: A Systematic Review
Coclite, D., Napoletano, A., Gianola, S., del Monaco, A., D’Angelo, D., Fauci, A. J., Iacorossi, L., Latina, R., Torre, G. L., Mastroianni, C. M., et al. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2020.594269 · Cited by 95
“We planned to use the PROBAST (Prediction model Risk Of Bias Assessment Tool) for Prediction Model Studies (29).”
Prognostic prediction models for chronic postsurgical pain in adults: a systematic review
Papadomanolakis‐Pakis, N., Uhrbrand, P., Haroutounian, S., & Nikolajsen, L. (2021) · Pain · doi:10.1097/j.pain.0000000000002261 · Cited by 99
Vernooij, L. M., van Klei, W. A., Moons, K. G. M., Takada, T., van Waes, J. A. R., & Damen, J. A. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013139.pub2 · Cited by 113
Self-citationStals, M. A. M., Takada, T., Kraaijpoel, N., van Es, N., Buller, H. R., Courtney, D. M., Freund, Y., Galipienzo, J., Gal, G. L., Ghanima, W., et al. (2021) · Annals of Internal Medicine · doi:10.7326/m21-2625 · Cited by 96
“5) 174 (18) 288 (30) 950 (411–1000) 229 (24) 258 (15) 127 (7.”
BackgroundMethodInfluentialSelf-citationBullock, G. S., Mylott, J. A., Hughes, T., Nicholson, K. F., Riley, R. D., & Collins, G. S. (2022) · Sports Medicine · doi:10.1007/s40279-022-01698-9 · Cited by 108
“Following PROBAST, studies are considered to be at high risk of bias overall when one or more appraisal domains are judged to be at high risk [32].”
InfluentialSelf-citationSolmi, M., Cortese, S., Vita, G., Prisco, M. D., Raduà, J., Dragioti, E., Köhler‐Forsberg, O., Madsen, N. M., Rohde, C., Eudave, L., et al. (2023) · Molecular Psychiatry · doi:10.1038/s41380-023-02298-3 · Cited by 105
“We conducted simulation analyses to test whether the criteria de fi ning MCPs were sensitive and speci fi c. Predictors were also classi fi ed into modi fi able or non-modi fi able, and patient, ilnness-, or treatment-related factors.”
MethodRamspek, C. L., Evans, M., Wanner, C., Drechsler, C., Chesnaye, N. C., Szymczak, M., Krajewska, M., Torino, C., Porto, G., Hayward, S., et al. (2021) · Journal of the American Society of Nephrology · doi:10.1681/asn.2020071077 · Cited by 94
Ara, J., & Khan, M. K. (2021) · American Journal of Interdisciplinary Studies · doi:10.63125/1p8gbp15 · Cited by 90
Garcia Santa Cruz, B., Bossa, M. N., Sölter, J., & Husch, A. D. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102225 · Cited by 89
Detection of Sleep Apnea Using Wearable AI: Systematic Review and Meta-Analysis
Abd-Alrazaq, A. A., Aslam, H., AlSaad, R., Alsahli, M. A., Ahmed, A., Damseh, R., Aziz, S., & Sheikh, J. I. (2024) · Journal of Medical Internet Research · doi:10.2196/58187 · Cited by 122
“This adaptation involved substituting some of the original criteria, which were not applicable to our context, with more relevant criteria from the Prediction Model Risk of Bias Assessment Tool [33].”
Mohan, G., & Chattopadhyay, S. K. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2020.1460 · Cited by 88
Predicting sepsis onset in ICU using machine learning models: a systematic review and meta-analysis
Yang, Z., Cui, X., & Song, Z. (2023) · BMC Infectious Diseases · doi:10.1186/s12879-023-08614-0 · Cited by 68
Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers
Semmelrock, H., Ross-Hellauer, T., Kopeinik, S., Theiler, D., Haberl, A., Thalmann, S., & Kowald, D. (2024) · arXiv · doi:10.48550/arxiv.2406.14325 · Cited by 88
Warfarin dosing algorithms: A systematic review
Asiimwe, I. G., Zhang, E. J., Osanlou, R., Jorgensen, A., & Pirmohamed, M. (2020) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.14608 · Cited by 93
“Although this tool focuses on prediction models that consider binary or time-to-event outcomes and studies that use generalized linear modelling, its authors encourage its use in studies that consider other outcomes and other machine learning techniques such as those explored in this review.(12) It should, however, be…”
MethodInfluentialGoncharov, M., Pisov, M., Shevtsov, A. E., Shirokikh, B., Kurmukov, A. G., Blokhin, I. A., Chernina, V., Solovev, A. V., Gombolevskiy, V. А., Morozov, S. P., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102054 · Cited by 97
Du, M., Haag, D. G., Lynch, J. W., & Mittinty, M. (2020) · Cancers · doi:10.3390/cancers12102802 · Cited by 95
Systematic review of prediction models for pulmonary tuberculosis treatment outcomes in adults
Peetluk, L. S., Ridolfi, F. M., Rebeiro, P. F., Liu, D., Rolla, V. C., & Sterling, T. R. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044687 · Cited by 99
Ma, J., Dhiman, P., Qi, C., Bullock, G. S., van Smeden, M., Riley, R. D., & Collins, P. G. S. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.07.017 · Cited by 84
“[61,62].”
Self-citationHow accurate are suicide risk prediction models? asking the right questions for clinical practice.
Whiting, D., & Fazel, S. (2019) · Evidence Based Mental Health · doi:10.1136/ebmental-2019-300102 · Cited by 83
“New rating scales should be used, such as PROBAST,10 which have been developed for prediction models.”
MethodInfluentialMiles, J., Turner, J., Jacques, R., Williams, J., & Mason, S. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00084-1 · Cited by 75
“Risk of bias and applicability was undertaken using the Prediction model Risk of Bias Assessment Tool (PROBAST) [22].”
MethodThe impact of climate change on the risk factors for tuberculosis: A systematic review
Kharwadkar, S., Attanayake, V., Duncan, J., Navaratne, N., & Benson, J. (2022) · Environmental Research · doi:10.1016/j.envres.2022.113436 · Cited by 88
Kee, O. T., Harun, H., Mustafa, N., Abdul Murad, N. A., Chin, S. F., Jaafar, R., & Abdullah, N. (2023) · Cardiovascular Diabetology · doi:10.1186/s12933-023-01741-7 · Cited by 82
Andaur Navarro, C. L., Damen, J. A. A. G., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., Moons, K. G., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-038832 · Cited by 83
Self-citationQuinn, T. J., Burton, J. K., Carter, B., Cooper, N. J., Dwan, K. M., Field, R., Freeman, S. C., Geue, C., Hsieh, P.-H., McGill, K., et al. (2021) · BMC Medicine · doi:10.1186/s12916-021-01920-x · Cited by 84
Chowdhury, M. Z. I., Naeem, I., Quan, H., Leung, A. A. C., Sikdar, K. C., O’Beirne, M., & Turin, T. C. (2022) · PLoS ONE · doi:10.1371/journal.pone.0266334 · Cited by 94
“When additional questions are added to the PROBAST, these questions need to be appropriately phrased, and specific guidance on assessing these signaling questions also needs to be provided [14, 15].”
BackgroundMethodInfluentialArina, P., Kaczorek, M. R., Hofmaenner, D. A., Pisciotta, W., Refinetti, P., Singer, M., Mazomenos, E. B., & Whittle, J. (2024) · Anesthesiology · doi:10.1097/aln.0000000000004764 · Cited by 81
“Bias was evaluated using the Prediction model Risk Of Bias Assessment Tool (PROBAST), 18 and an ad hoc classification was developed to determine the readiness level of the machine learning algorithms reported.”
Jimenez‐Solem, E., Petersen, T. S., Hansen, C., Hansen, C., Lioma, C., Igel, C., Boomsma, W., Krause, O., Lorenzen, S. S., Selvan, R., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-81844-x · Cited by 92
Groot, O. Q., Ogink, P. T., Lans, A., Twining, P. K., Kapoor, N. D., DiGiovanni, W., Bindels, B. J. J., Bongers, M. E. R., Oosterhoff, J. H. F., Karhade, A. V., et al. (2022) · Journal of Orthopaedic Research · doi:10.1002/jor.25036 · Cited by 80
“PROBAST assesses the risk of bias in prognostic prediction model studies.(5) This tool consists of 20 signaling questions across four domains: participants selection (1), predictors (2), outcome (3), and analysis (4).”
MethodArtificial Intelligence for Large-Vessel Occlusion Stroke: A Systematic Review
Shlobin, N. A., Baig, A. A., Waqas, M., Patel, T. R., Dossani, R. H., Wilson, M., Cappuzzo, J. M., Siddiqui, A. H., Tutino, V. M., & Levy, E. I. (2021) · World Neurosurgery · doi:10.1016/j.wneu.2021.12.004 · Cited by 89
A critical moment in machine learning in medicine: on reproducible and interpretable learning
Ciobanu-Caraus, O., Aicher, A., Kernbach, J. M., Regli, L., Serra, C., & Staartjes, V. E. (2024) · Acta Neurochirurgica · doi:10.1007/s00701-024-05892-8 · Cited by 97
“A systematic review of bias assessments analyzing over 2000 clinical prediction models found that a substantial proportion of these—ranging from one quarter to two thirds— displayed a high risk of bias based on either their statistical analysis, outcome definition, or participant selection [3, 66].”
Guni, A., Sounderajah, V., Whiting, P. F., Bossuyt, P. M., Darzi, A. W., & Ashrafian, H. (2024) · JMIR Research Protocols · doi:10.2196/58202 · Cited by 86
Self-citationFazel, S., Burghart, M., Fanshawe, T. R., Gil, S. D., Monahan, J. T., & Yu, R. (2022) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2022.101902 · Cited by 98
“The risk of bias within each study was assessed with the Prediction model Risk Of bias ASsessment Tool (PROBAST; Wolff et al., 2019).”
BackgroundMethodInfluentialArtificial intelligence in ovarian cancer histopathology: a systematic review
Breen, J., Allen, K., Zucker, K., Adusumilli, P., Scarsbrook, A. F., Hall, G. D., Orsi, N. M., & Ravikumar, N. (2023) · npj Precision Oncology · doi:10.1038/s41698-023-00432-6 · Cited by 86
“The PROBAST applicability of research analysis was not implemented as it is unsuitable for such a diverse array of possible research questions.”
InfluentialRisk of bias of prognostic models developed using machine learning: a systematic review in oncology
Dhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., et al. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00126-w · Cited by 79
“The primary outcome was risk of bias for each developed model and was assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST) [30, 31].”
BackgroundMethodSelf-citationHaines, K. J., Hibbert, E., McPeake, J. M., Anderson, B. J., Bienvenu, O. J., Andrews, A., Brummel, N. E., Ferrante, L. E., Hopkins, R. O., Hough, C. L., et al. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004659 · Cited by 80
“) reviewed each article, using the Prediction model Risk Of Bias Assessment Tool (PROBAST) (14) to evaluate risk of bias for the included studies.”
InfluentialPfob, A., Mehrara, B. J., Nelson, J. A., Wilkins, E. G., Pusic, A. L., & Sidey-Gibbons, C. J. (2021) · Annals of Surgery · doi:10.1097/sla.0000000000004862 · Cited by 86
The role of AI in emergency department triage: An integrative systematic review
Arab, R. A. E., & Moosa, O. A. A. (2025) · Intensive and Critical Care Nursing · doi:10.1016/j.iccn.2025.104058 · Cited by 96
Issaiy, M., Zarei, D., & Saghazadeh, A. (2023) · World Journal of Emergency Surgery · doi:10.1186/s13017-023-00527-2 · Cited by 96
“The studies included in the research were evaluated using the Prediction Model Risk of Bias Assessment Tool (PROBAST), a tool designed to assess the risk of bias across four distinct domains while also evaluating the applicability of diagnostic and prognostic models within the research context [18].”
MethodDelta radiomics: an updated systematic review
Nardone, V., Reginelli, A., Rubini, D., Gagliardi, F., Tufo, S. D., Belfiore, M. P., Boldrini, L., Desideri, I., & Cappabianca, S. (2024) · La radiologia medica · doi:10.1007/s11547-024-01853-4 · Cited by 86
“These include the use of checklists for both radiomics and artificial intelligence [93, 94], the stand-ardization of image biomarkers [95], and tools designed to assess the risk of bias and the applicability of radiomics studies [96].”
MethodGroot, O. Q., Bindels, B. J. J., Ogink, P. T., Kapoor, N. D., Twining, P. K., Collins, A. K., Bongers, M. E. R., Lans, A., Oosterhoff, J. H. F., Karhade, A. V., et al. (2021) · Acta Orthopaedica · doi:10.1080/17453674.2021.1910448 · Cited by 82
Koçak, B. (2022) · Diagnostic and Interventional Radiology · doi:10.5152/dir.2022.211297 · Cited by 85
“112 Among them, CLAIM is specifically designed for medical imaging-related AI.”
BackgroundMartínez-Millana, A., Saez-Saez, A., Tornero-Costa, R., Muscat, N. A., Traver, V., & Novillo-Ortiz, D. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104855 · Cited by 81
Brave New World of Artificial Intelligence: Its Use in Antimicrobial Stewardship—A Systematic Review
Pinto-de-Sá, R., Sousa‐Pinto, B., & Costa‐de‐Oliveira, S. (2024) · Antibiotics · doi:10.3390/antibiotics13040307 · Cited by 79
Alabi, R. O., Elmusrati, M. S., Leivo, I. V. V., Almangush, A., & Mäkitie, A. A. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105464 · Cited by 75
de Jong, Y., Ramspek, C. L., Zoccali, C., Jager, K. J., Dekker, F. W., & van Diepen, M. (2021) · Nephrology · doi:10.1111/nep.13913 · Cited by 77
Pfob, A., Sidey-Gibbons, C. J., Lee, H., Tasoulis, M. K., Koelbel, V., Golatta, M., Rauch, G. M., Smith, B. D., Valero, V., Han, W., et al. (2020) · European Journal of Cancer · doi:10.1016/j.ejca.2020.11.006 · Cited by 74
“We used the PROBAST tool [39] to assess the risk of bias and the applicability of our model.”
Rhon, D. I., Molloy, J. M., Monnier, A., Hando, B., & Newman, P. (2021) · European Journal of Sport Science · doi:10.1080/17461391.2021.1931464 · Cited by 80
“We used the Prediction model Risk Of Bias ASsessment Tool (PROBAST), which has four domains (participants, predictors, outcome, and analysis) (Wolff et al., 2019) to assess the methodological quality of each study.”
MethodGallifant, J., Zhang, J., del Pilar Arias Lopez, M., Zhu, T., Camporota, L., Celi, L. A., & Formenti, F. (2022) · British Journal of Anaesthesia · doi:10.1016/j.bja.2021.09.025 · Cited by 72
Lawson, D. O., Puljak, L., Pieper, D., Schandelmaier, S., Collins, G. S., Brignardello‐Petersen, R., Moher, D., Tugwell, P., Welch, V. A., Samaan, Z., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-040478 · Cited by 73
The Internet of Federated Things (IoFT)
Kontar, R., Shi, N., Yue, X., Chung, S., Byon, E., Chowdhury, M., Jin, J., Kontar, W., Masoud, N., Nouiehed, M., et al. (2021) · IEEE Access · doi:10.1109/access.2021.3127448 · Cited by 66
“While the medical literature contains numerous examples of apparently high-performing models, many of these studies suffer from poor generalization, which can either be demonstrated through an independent examination of its methods (by applying the PROBAST tool [299]) or through external validation of the findings.”
Methodvan Daalen, K. R., Zhang, D., Kaptoge, S. K., Paige, E., Angelantonio, E. D., & Pennells, L. A. M. (2024) · The Lancet Global Health · doi:10.1016/s2214-109x(24)00210-9 · Cited by 87
Predicting relapse or recurrence of depression: systematic review of prognostic models
Moriarty, A. S., Meader, N., Snell, K. I. E., Riley, R. D., Paton, L. W., Dawson, S., Hendon, J., Chew-Graham, C. A., Gilbody, S., Churchill, R., et al. (2022) · The British Journal of Psychiatry · doi:10.1192/bjp.2021.218 · Cited by 71
Self-citationShan, G., Chen, X., Wang, C., Liu, L., Gu, Y., Jiang, H., & Shi, T. (2025) · JMIR Medical Informatics · doi:10.2196/64963 · Cited by 71
“The Prediction Model Risk of Bias Assessment Tool (PROBAST) was used to evaluate the risk of bias and applicability of all included studies [8].”
Predicting falls in community-dwelling older adults: a systematic review of prognostic models
Gade, G. V., Jørgensen, M. G., Ryg, J., Riis, J., Thomsen, K., Masud, T., & Andersen, S. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044170 · Cited by 74
Heus, P., Reitsma, J. B., Collins, G. S., Damen, J. A. A. G., Scholten, R. J. P. M., Altman, D. G., Moons, K. G. M., & Hooft, L. (2020) · Annals of Internal Medicine · doi:10.7326/m20-0193 · Cited by 70
“Reporting this essential step in the abstract helps the reader judge the study's risk of bias (13, 14).”
BackgroundSelf-citationMachine learning techniques for mortality prediction in emergency departments: a systematic review
Naemi, A., Schmidt, T., Mansourvar, M., Naghavi-Behzad, M., Ebrahimi, A., & Wiil, U. K. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-052663 · Cited by 64
Predicting Patient Deterioration: A Review of Tools in the Digital Hospital Setting
Mann, K. D., Good, N. M., Fatehi, F., Khanna, S., Campbell, V. K., Conway, R., Sullivan, C. M., Staib, A. N., Joyce, C., & Cook, D. (2021) · Journal of Medical Internet Research · doi:10.2196/28209 · Cited by 79
“An assessment of study quality, including risks of bias, was conducted using PROBAST (Prediction Model Risk of Bias Assessment Tool; UMC Utrecht) [21] and ROBINS-I (Risk of Bias in Non-Randomised Intervention Studies; Cochrane) tool [22].”
Fisher, V., Li, W. W., & Malabu, U. H. (2023) · Applied Psychology Health and Well-Being · doi:10.1111/aphw.12441 · Cited by 73
Strategies to improve the impact of artificial intelligence on health equity: scoping review
Berdahl, C. T., Baker, L., Mann, S., Osoba, O., & Girosi, F. (2023) · JMIR AI · doi:10.2196/42936 · Cited by 68
“AI: artificial intelligence BeHEMoTh: Behavior, Health condition, Exclusions, Models, or Theories FDA: Food and Drug Administration IEEE: Institute of Electrical and Electronics Engineers PRISMA-ScR: Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews PROBAST: Prediction Mo…”
MethodInfluentialSuri, J. S., Agarwal, S., Gupta, S., Puvvula, A., Viskovic, K., Suri, N., Alizad, A., El-Baz, A., Saba, L., Fatemi, M., et al. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3103839 · Cited by 65
“On the contrary, ROBINS-I [99] and PROBAST [100] helps in identification of factors that contribute to RoB.”
Clinical prediction models and the multiverse of madness
Riley, R. D., Pate, A., Dhiman, P., Archer, L., Martin, G. P., & Collins, G. S. (2023) · BMC Medicine · doi:10.1186/s12916-023-03212-y · Cited by 87
“…whether a model’s predictions are likely reliable at the individual level; enable more informed judgements about the model’s quality (risk of bias [36]) in the context of its potential use; identify whether larger development datasets (or external information) are needed to reduce instability;…”
BackgroundSelf-citationFreedman, B., Li, W. W., Liang, Z., Hartin, P., & Biedermann, N. (2024) · Journal of Advanced Nursing · doi:10.1111/jan.16111 · Cited by 71
Uncertainty of risk estimates from clinical prediction models: rationale, challenges, and approaches
Riley, R. D., Collins, G. S., Kirton, L., Snell, K. I. E., Ensor, J., Whittle, R., Dhiman, P., van Smeden, M., Liu, X., Alderman, J. E., et al. (2025) · BMJ · doi:10.1136/bmj-2024-080749 · Cited by 106
“16 When quantifying uncertainty, datasets should be used that are high quality (eg, with appropriate methods of measuring outcomes and predictors) 37 and representative of the target population and setting where the model will be applied in practice.”
Self-citationAI Reporting Guidelines: How to Select the Best One for Your Research
Klontzas, M. E., Gatti, A. A., Tejani, A. S., & Kahn, C. E. (2023) · Radiology: Artificial Intelligence · doi:10.1148/ryai.230055 · Cited by 66
Wu, S., Zeng, N., Sun, F., Zhou, J., Wu, X., Sun, Y., Wang, B., Zhan, S., Kong, Y., Jia, J., et al. (2021) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2021.02.040 · Cited by 71
Cong, B., Dighero, I., Zhang, T., Chung, A., Nair, H., & Li, Y. (2023) · BMC Medicine · doi:10.1186/s12916-023-02932-5 · Cited by 71
“Quality assessment We developed a modified quality assessment checklist based on the PROBAST risk of bias assessment tool for model studies [13].”
MethodMachine Learning: A New Approach for Dose Individualization
Li, Q., Tang, B., Wu, Y., Yao, B., Zhang, W., Zheng, Y., Zhou, Y., van den Anker, J. N., Hao, G., & Zhao, W. (2023) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.3049 · Cited by 70
“9,10 Extracted data included the study drugs, study population, data source, sample size, predictive target, predictor, details of the ML model development (e.g., ML algorithms and feature selection), validation type, and predictive performance measurement.”
Artificial intelligence bias in medical system designs: a systematic review
Kumar, A., Aelgani, V., Vohra, R., Gupta, S. K., Bhagawati, M., Paul, S., Saba, L., Suri, N., Khanna, N. N., Laird, J. R., et al. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16029-x · Cited by 65
Machine-Learning Applications in Oral Cancer: A Systematic Review
López-Cortés, X. A., Matamala, F., Venegas, B., & Rivera, C. (2022) · Applied Sciences · doi:10.3390/app12115715 · Cited by 67
“The findings of the ROB assessment were assessed in this fashion using PROBAST, as indicated in Table 1.”
MethodInfluential
Metadata
- Authors
- Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group†
- Affiliations
- Kleijnen Systematic Reviews (United Kingdom) (GB)
- Utrecht University (NL)
- Keele University (GB)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom (G.S.C.)
- Maastricht University (NL)
- Institute of Applied Health Research, National Institute for Health Research Birmingham Biomedical Research Centre, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom (S.M.)
- ORCID
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2019-01-01
- Volume
- 170
- Issue
- 1
- Pages
- 51-58
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W2907554860
- Cited by
- 2,984
- References
- 42
- Citations by year
- 2026: 921 · 2025: 561 · 2024: 416 · 2023: 343 · 2022: 303 · 2021: 250 · 2020: 140 · 2019: 45
- FWCI
- 83.39
- Type
- journal article
- Language
- en
- Open access
- Licence
- other-oa
- www.acpjournals.org/journal/aim/text-and-data-mining (tdm)
- Locations
- Annals of Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- Bristol Research (University of Bristol) (acceptedVersion, other-oa)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- FreiDok plus (Universitätsbibliothek Freiburg) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- Radboud Repository (Radboud University) (submittedVersion)
- Explore Bristol Research (submittedVersion, other-oa)
- Apollo (University of Cambridge) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Apollo
- Funders
- National Institute for Health Research Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (ZONMW 918.10.615, ZONMW 912.08.004)
- National Institute for Health Research School for Primary Care Research (390)
- National Institute for Health Research Biomedical Research Centre, Oxford
- National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) West at University Hospitals Bristol NHS Foundation Trust
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- PROBAST, risk of bias assessment, diagnostic model, prognostic model, applicability, critical appraisal, reporting guidelines, Delphi consensus, systematic reviews, personalized prediction
- MeSH terms
- Systematic Reviews as Topic, Decision Support Techniques, Delphi Technique, Diagnosis, Humans, Prognosis, Research Design, Models, Statistical, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.