PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration
Open accessThe DOI matches its Crossref record; 194 of the 201 references it cites were found in a registry or an index.
Abstract
Prediction models in health care use predictors to estimate for an individual the probability that a condition or disease is already present (diagnostic model) or will occur in the future (prognostic model). Publications on prediction models have become more common in recent years, and competing prediction models frequently exist for the same outcome or target population. Health care providers, guideline developers, and policymakers are often unsure which model to use or recommend, and in which persons or settings. Hence, systematic reviews of these studies are increasingly demanded, required, and performed. A key part of a systematic review of prediction models is examination of risk of bias and applicability to the intended population and setting. To help reviewers with this process, the authors developed PROBAST (Prediction model Risk Of Bias ASsessment Tool) for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic. PROBAST was developed through a consensus process involving a group of experts in the field. It includes 20 signaling questions across 4 domains (participants, predictors, outcome, and analysis). This explanation and elaboration document describes the rationale for including each domain and signaling question and guides researchers, reviewers, readers, and guideline developers in how to use them to assess risk of bias and applicability concerns. All concepts are illustrated with published examples across different topics. The latest version of the PROBAST checklist, accompanying documents, and filled-in examples can be downloaded from www.probast.org.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Self-citationROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Self-citationQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349
Self-citationMeasuring the Accuracy of Diagnostic Systems
Swets, J. A. (1988) · Science · doi:10.1126/science.3287615 · Cited by 10,058
Multiple imputation using chained equations: Issues and guidance for practice
White, I. R., Royston, P., & Wood, A. M. (2011) · Statistics in Medicine · doi:10.1002/sim.4067 · Cited by 9,792
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655
Research works printed but not linked: 2
- 10.1177/0962280218785504.30032705 · doi:10.1177/0962280218785504.30032705
- Wong SS, 2003, AMIA Annu Symp Proc, 728
Cited by
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
“Readers are referred to PROBAST 90 91 and the forthcoming PROBAST+AI 79 to assess the quality and risk of bias of prediction models (https://www.probast.org/).”
BackgroundSelf-citationWynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Self-citationPROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group† (2019) · Annals of Internal Medicine · doi:10.7326/m18-1376 · Cited by 2,984
Self-citationCalculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917
Self-citationNagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152
Moons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-082505 · Cited by 1,069
Self-citationCollins, G. S., Dhiman, P., Andaur Navarro, C. L., Ma, J., Hooft, L., Reitsma, J. B., Logullo, P., Beam, A. L., Peng, L., Van Calster, B., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-048008 · Cited by 852
Self-citationEvaluation of clinical prediction models (part 2): how to undertake an external validation study
Riley, R. D., Archer, L., Snell, K. I. E., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L. J., & Collins, G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074820 · Cited by 401
“…predicted values for each Box 1: Signalling questions within the first three domains of PROBAST (Prediction model Risk Of Bias ASsessment Tool) 19 20 that are important to consider when ensuring a dataset for external validation is fit for purpose • Was the outcome defined and determined in a…”
BackgroundSelf-citationAndaur Navarro, C. L., Damen, J. A. A., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., Moons, K. G. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2281 · Cited by 415
“19 20 Using a prediction model considered at high risk of bias might lead to unnecessary or insufficient interventions and thus affect patients’ health and health systems.”
BackgroundMethodSelf-citation(2024) · BMJ · doi:10.1136/bmj.q902 · Cited by 416
“Readers are referred to PROBAST 90 91 and the forthcoming PROBAST+AI 79 to assess the quality and risk of bias of prediction models (https://www.probast.org/).”
Designing deep learning studies in cancer diagnostics
Kleppe, A., Skrede, O.-J., De Raedt, S., Liestøl, K., Kerr, D. J., & Danielsen, H. E. (2021) · Nature Reviews Cancer · doi:10.1038/s41568-020-00327-9 · Cited by 316
AI pitfalls and what not to do: mitigating bias in AI
Gichoya, J. W., Thomas, K., Celi, L. A., Safdar, N., Banerjee, I., Banja, J. D., Seyyed-Kalantari, L., Trivedi, H., & Purkayastha, S. (2023) · The British Journal of Radiology · doi:10.1259/bjr.20230023 · Cited by 277
van de Sande, D., van Genderen, M. E., Huiskens, J., Gommers, D., & van Bommel, J. (2021) · Intensive Care Medicine · doi:10.1007/s00134-021-06446-7 · Cited by 279
Leisman, D. E., Harhay, M. O., Lederer, D. J., Abramson, M., Adjei, A. A., Bakker, J., Ballas, Z. K., Barreiro, E., Bell, S. C., Bellomo, R., et al. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004246 · Cited by 271
Cowley, L. E., Farewell, D. M., Maguire, S., & Kemp, A. M. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0060-y · Cited by 263
Clinical prediction models: diagnosis versus prognosis
van Smeden, M., Reitsma, J. B., Riley, R. D., Collins, G. S., & Moons, K. G. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.01.009 · Cited by 238
Self-citationEvaluation framework to guide implementation of AI systems into healthcare settings
Reddy, S., Rogers, W., Makinen, V.-P., Coiera, E., Brown, P., Wenzel, M., Weicken, E., Ansari, S., Mathur, P., Casey, A., et al. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100444 · Cited by 230
Damen, J. A., Pajouheshnia, R., Heus, P., Moons, K. G. M., Reitsma, J. B., Scholten, R. J. P. M., Hooft, L., & Debray, T. P. A. (2019) · BMC Medicine · doi:10.1186/s12916-019-1340-7 · Cited by 250
“Risk of bias was assessed based on a combination of the CHARMS checklist [23] and a preliminary version of the Cochrane Prediction study Risk Of Bias Assessment Tool (PROBAST) [27, 28] (Additional file 5).”
MethodSelf-citationSalazar de Pablo, G., Studerus, E., Vaquerizo-Serrano, J., Irving, J., Catalan, A., Oliver, D., Baldwin, H., Danese, A., Fazel, S., Steyerberg, E. W., et al. (2021) · Schizophrenia Bulletin · doi:10.1093/schbul/sbaa120 · Cited by 219
Artificial Intelligence and Liability in Medicine: Balancing Safety and Innovation
MALIHA, G., GERKE, S., COHEN, I. G., & PARIKH, R. B. (2021) · The Milbank Quarterly · doi:10.1111/1468-0009.12504 · Cited by 211
Snell, K. I. E., Levis, B., Damen, J. A. A., Dhiman, P., Debray, T. P. A., Hooft, L., Reitsma, J. B., Moons, K. G. M., Collins, G. S., & Riley, R. D. (2023) · BMJ · doi:10.1136/bmj-2022-073538 · Cited by 190
“Support 24 Describe sources of financial or non-financial support for the review, and the role of the funders or sponsors in the review.”
BackgroundSelf-citationMinimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198
Self-citationBellou, V., Belbasis, L., Konstantinidis, A. K., Tzoulaki, I., & Evangelou, E. (2019) · BMJ · doi:10.1136/bmj.l5358 · Cited by 197
Artificial Intelligence for COVID-19: A Systematic Review
Wang, L., Zhang, Y., Wang, D., Tong, X., Liu, T., Zhang, S., Huang, J., Zhang, L., Chen, L., Fan, H., et al. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.704256 · Cited by 182
Identifying predictors of ventral hernia recurrence: systematic review and meta-analysis
Parker, S. G., . Mallett, S., Quinn, L., Wood, C. P. J., Boulton, R. W., Jamshaid, S., . Erotocritou, M., . Gowda, S., . Collier, W., Plumb, A. A. O., et al. (2021) · BJS Open · doi:10.1093/bjsopen/zraa071 · Cited by 163
Self-citationArtificial Intelligence for COVID-19: Rapid Review
(2020) · doi:10.2196/21476 · Cited by 160
Barreñada, L., Dhiman, P., Timmerman, D., Boulesteix, A., & CALSTER, B. V. (2024) · Diagnostic and Prognostic Research · doi:10.1186/s41512-024-00177-1 · Cited by 216
“First, it is well known that apparent performance, i.e., performance assessment on the exact same dataset that was used to train the model, is overly optimistic [30, 31].”
BackgroundDretzke, J., Chuchu, N., Agarwal, R., Herd, C. P., Chua, W., Fabritz, L., Bayliss, S. E., Kotecha, D., Deeks, J. J., Kirchhof, P. F., et al. (2020) · EP Europace · doi:10.1093/europace/euaa041 · Cited by 160
Drukker, K., Chen, W., Gichoya, J. W., Gruszauskas, N. P., Kalpathy-Cramer, J., Koyejo, S., Myers, K. J., Sá, R. C., Sahiner, B., Whitney, H. M., et al. (2023) · Journal of Medical Imaging · doi:10.1117/1.jmi.10.6.061104 · Cited by 149
Risk factors for overuse injuries in short- and long-distance running: A systematic review
van Poppel, D., van der Worp, M. P., Slabbekoorn, A., van den Heuvel, S. P., van Middelkoop, M., Koes, B. W., Verhagen, A. P., & Scholten-Peeters, G. G. M. (2020) · Journal of sport and health science/Journal of Sport and Health Science · doi:10.1016/j.jshs.2020.06.006 · Cited by 178
Sufriyana, H., Husnayain, A., Chen, Y.-L., Kuo, C.-Y., Singh, O., Yeh, T., Wu, Y.-W., & Su, E. (2019) · JMIR Medical Informatics · doi:10.2196/16503 · Cited by 116
den Bosch, L. V., van der Schaaf, A., van der Laan, H. P., Hoebers, F. J. P., Wijers, O. B., van den Hoek, J. G. M., Moons, K. G. M., Reitsma, J. B., Steenbakkers, R. J. H. M., Schuit, E., et al. (2021) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2021.01.024 · Cited by 144
Self-citationDhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., et al. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01577-x · Cited by 140
Self-citationThe Effectiveness of Artificial Intelligence in Detection of Oral Cancer
Al-Rawi, N. H., Sultan, A. S., Rajai, B., Shuaeeb, H., Alnajjar, M., Alketbi, M. M., Mohammad, Y., Shetty, S. R., & Mashrah, M. A. (2022) · International Dental Journal · doi:10.1016/j.identj.2022.03.001 · Cited by 153
The current status of risk-stratified breast screening
Clift, A. K., Dodwell, D. J., Lord, S., Petrou, S., Brady, M., Collins, P. G. S., & Hippisley‐Cox, J. (2021) · British Journal of Cancer · doi:10.1038/s41416-021-01550-3 · Cited by 155
Clinical applications of artificial intelligence in sepsis: A narrative review
Schinkel, M., Paranjape, K., Nannan Panday, R. S., Skyttberg, N., & Nanayakkara, P. W. B. (2019) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2019.103488 · Cited by 130
Pandor, A., Tonkins, M., Goodacre, S. W., Sworn, K., Clowes, M., Griffin, X. L., Holland, M., Hunt, B. J., de Wit, K., & Horner, D. E. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-045672 · Cited by 144
Campagnini, S., Arienti, C., Patrini, M., Liuzzi, P., Mannini, A., & Carrozza, M. C. (2022) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-022-01032-4 · Cited by 142
Pan, P., Li, Y., Xiao, Y., Han, B., Su, L., Su, M., Li, Y., Zhang, S., Jiang, D., Chen, X., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/23128 · Cited by 129
Embryo selection with artificial intelligence: how to evaluate and compare methods?
Kragh, M. F., & Karstoft, H. (2021) · Journal of Assisted Reproduction and Genetics · doi:10.1007/s10815-021-02254-6 · Cited by 135
A scoping review of artificial intelligence-based methods for diabetes risk prediction
Mohsen, F., Al-Absi, H. R. H., Yousri, N. A., El Hajj, N., & Shah, Z. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00933-5 · Cited by 125
“Moreover, hold-out validation presents a concern as it reduces the sample size available for model development and may not utilize the data effectively 59,60 .”
MethodFernandez-Felix, B. M., López‐Alcalde, J., Figuls, M. R. I., Muriel, A., & Zamora, J. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-01849-0 · Cited by 127
“The PROBAST tool (Prediction model Risk Of Bias Assessment Tool) is a checklist for assessing the risk of bias and the applicability of prognostic model studies [6, 7].”
MethodLarrue, H., D’Amico, G., Olivas, P., Lv, Y., Bucsics, T., Rudler, M., Sauerbruch, T., Hernández–Gea, V., Han, G., Reiberger, T., et al. (2023) · Journal of Hepatology · doi:10.1016/j.jhep.2023.04.028 · Cited by 135
The trans-omics landscape of COVID-19
Wu, P., Chen, D., Ding, W., Wu, P., Hou, H., Bai, Y., Zhou, Y., Li, K., Xiang, S., Liu, P., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-24482-1 · Cited by 131
Machine Learning Prediction Models for Gestational Diabetes Mellitus: Meta-analysis
Zhang, Z., Yang, L., Han, W., Wu, Y., Zhang, L., Gao, C., Jiang, K., Liu, Y., & Wu, H. (2021) · Journal of Medical Internet Research · doi:10.2196/26634 · Cited by 138
Andaur Navarro, C. L., Damen, J. A. A., van Smeden, M., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., et al. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.11.015 · Cited by 121
Self-citationPredicting breast cancer 5-year survival using machine learning: A systematic review
Li, J., Zhou, Z., Dong, J., Fu, Y., Li, Y., Luan, Z., & Peng, X. (2021) · PLoS ONE · doi:10.1371/journal.pone.0250370 · Cited by 138
Smith, L. A., Oakden-Rayner, L., Bird, A., Zeng, M., To, M.-S., Mukherjee, S., & Palmer, L. J. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00177-2 · Cited by 116
Buchlak, Q. D., Esmaili, N., Leveque, J., Bennett, C. C., Farrokhi, F. R., & Piccardi, M. (2021) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2021.04.043 · Cited by 123
How to conduct a systematic review and meta-analysis of prognostic model studies
Damen, J. A., Moons, K. G. M., van Smeden, M., & Hooft, L. (2022) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2022.07.019 · Cited by 123
Self-citationDhiman, P., Ma, J., Navarro, C., Speich, B., Bullock, G., Damen, J., Kirtley, S., Hooft, L., Riley, R., van Calster, B., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.024 · Cited by 94
Self-citationA survey of recent methods for addressing AI fairness and bias in biomedicine
Yang, Y., Lin, M., Zhao, H., Peng, Y., Huang, F., & Lu, Z. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104646 · Cited by 144
Porto, B. M. (2024) · BMC Emergency Medicine · doi:10.1186/s12873-024-01135-2 · Cited by 162
“This assessment examined four key domains: participants, predictors, outcomes, and analysis, with guidelines for implementation and interpretation provided in [88].”
We need to talk about nonprobability samples
Boyd, R. J., Powney, G. D., & Pescott, O. L. (2023) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2023.01.001 · Cited by 130
Pillay, J., Donovan, L., Guitard, S., Zakher, B., Gates, M., Gates, A., Vandermeer, B., Bougatsos, C., Chou, R., & Hartling, L. (2021) · JAMA · doi:10.1001/jama.2021.10404 · Cited by 119
Dhiman, P., Ma, J., Qi, C., Bullock, G. S., Sergeant, J. C., Riley, R. D., & Collins, P. G. S. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-02008-1 · Cited by 115
“Future research and recommendations Sample size is one of the minimum reporting recommendations in the TRIPOD reporting guideline and signalling question for PROBAST the risk of bias tool for prediction model studies [1, 2, 34, 35].”
BackgroundMethodSelf-citationClift, A. K., Dodwell, D., Lord, S., Petrou, S., Brady, M., Collins, G. S., & Hippisley-Cox, J. (2023) · BMJ · doi:10.1136/bmj-2022-073800 · Cited by 97
QUAPAS: An Adaptation of the QUADAS-2 Tool to Assess Prognostic Accuracy Studies
Lee, J., Mulder, F. I., Leeflang, M., Wolff, R. F., Whiting, P. F., & Bossuyt, P. M. (2022) · Annals of Internal Medicine · doi:10.7326/m22-0276 · Cited by 111
Self-citationSoda, P., D’Amico, N. C., Tessadori, J., Valbusa, G., Guarrasi, V., Bortolotto, C., Akbar, M. U., Sicilia, R., Cordelli, E., Fazzini, D., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102216 · Cited by 104
Sufriyana, H., Wu, Y., & Su, E. C. (2020) · EBioMedicine · doi:10.1016/j.ebiom.2020.102710 · Cited by 112
Model-based Prediction of Critical Illness in Hospitalized Patients with COVID-19
Schalekamp, S., Huisman, M., van Dijk, R. A., Boomsma, M. F., Jorge, P. F., de Boer, W. S., Herder, G. J. M., Bonarius, M., Groot, O. A., Jong, E., et al. (2020) · Radiology · doi:10.1148/radiol.2020202723 · Cited by 105
Debray, T. P. A., Collins, P. G. S., Riley, R. D., Snell, K. I. E., CALSTER, B. V., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071018 · Cited by 127
Self-citationMachine learning models for predicting preeclampsia: a systematic review
Ranjbar, A., Montazeri, F., Ghamsari, S. R., Mehrnoush, V., Roozbeh, N., & Darsareh, F. (2024) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-023-06220-1 · Cited by 127
“The PROBAST [11], consisting of 20 signaling questions divided into four domains (participants, predictors, outcome, and analysis), was used to assess the risk of bias and applicability of each included study.”
MethodPregnancy outcome predictors in antiphospholipid syndrome: A systematic review and meta-analysis
Walter, I. J., Haneveld, M. J. K., Lely, A. T., Bloemenkamp, K. W. M., Limper, M., & Kooiman, J. (2021) · Autoimmunity Reviews · doi:10.1016/j.autrev.2021.102901 · Cited by 105
Evaluating risk prediction models for adults with heart failure: A systematic literature review
Tanna, G. L. D., Wirtz, H. S., Burrows, K. L., & Globe, G. A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0224135 · Cited by 104
A systematic review on AI/ML approaches against COVID-19 outbreak
Dogan, O., Tiwari, S., Jabbar, M. A., & Guggari, S. (2021) · Complex & Intelligent Systems · doi:10.1007/s40747-021-00424-8 · Cited by 87
Mohan, G., & Chattopadhyay, S. K. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2020.1460 · Cited by 88
Vernooij, L. M., van Klei, W. A., Moons, K. G. M., Takada, T., van Waes, J. A. R., & Damen, J. A. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013139.pub2 · Cited by 113
Self-citationStals, M. A. M., Takada, T., Kraaijpoel, N., van Es, N., Buller, H. R., Courtney, D. M., Freund, Y., Galipienzo, J., Gal, G. L., Ghanima, W., et al. (2021) · Annals of Internal Medicine · doi:10.7326/m21-2625 · Cited by 92
Self-citationPrognostic prediction models for chronic postsurgical pain in adults: a systematic review
Papadomanolakis‐Pakis, N., Uhrbrand, P., Haroutounian, S., & Nikolajsen, L. (2021) · Pain · doi:10.1097/j.pain.0000000000002261 · Cited by 94
Using Machine Learning Technologies in Pressure Injury Management: Systematic Review
Jiang, M., Ma, Y., Guo, S., Jin, L., Lv, L., Han, L., & An, N. (2021) · JMIR Medical Informatics · doi:10.2196/25704 · Cited by 92
Kusuma, K., Larsen, M. E., Quiroz, J. C., Gillies, M. B., Burnett, A., Qian, J., & Torok, M. H. (2022) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2022.09.050 · Cited by 101
Prediction models for childhood asthma: A systematic review
Kothalawala, D. M., Kadalayil, L., Weiss, V. B. N., Kyyaly, A., Arshad, S. H., Holloway, J. W., & Rezwan, F. I. (2020) · Pediatric Allergy and Immunology · doi:10.1111/pai.13247 · Cited by 96
Navarro, C. A., Damen, J., Takada, T., Nijman, S., Dhiman, P., Ma, J., Collins, G., Bajpai, R., Riley, R., Moons, K., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-038832 · Cited by 49
Self-citationSystematic review of prediction models for pulmonary tuberculosis treatment outcomes in adults
Peetluk, L. S., Ridolfi, F. M., Rebeiro, P. F., Liu, D., Rolla, V. C., & Sterling, T. R. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044687 · Cited by 99
Gao, X., Cai, X., Yang, Y., Zhou, Y., & Zhu, W. (2021) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2021.757087 · Cited by 82
Arina, P., Kaczorek, M. R., Hofmaenner, D. A., Pisciotta, W., Refinetti, P., Singer, M., Mazomenos, E. B., & Whittle, J. (2024) · Anesthesiology · doi:10.1097/aln.0000000000004764 · Cited by 81
Warfarin dosing algorithms: A systematic review
Asiimwe, I. G., Zhang, E. J., Osanlou, R., Jorgensen, A., & Pirmohamed, M. (2020) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.14608 · Cited by 86
Mureșanu, S., Almăşan, O. C., Hedesiu, M. C., Dioşan, L. S., Dinu, C. M., & Jacobs, R. (2022) · Oral Radiology · doi:10.1007/s11282-022-00660-9 · Cited by 89
“According to the PROBAST assessment tool, out of the 59 included studies, 52 papers were at high risk of bias (ROB), two papers were rated as low ROB, and four papers as unclear.”
BackgroundMethodInfluentialRisk of bias of prognostic models developed using machine learning: a systematic review in oncology
Dhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., et al. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00126-w · Cited by 79
Self-citationLiu, W., Laranjo, L., Klimis, H., Chiang, J., Yue, J., Marschner, S. L., Quiroz, J. C., Jorm, L. R., & Chow, C. K. (2023) · European Heart Journal - Quality of Care and Clinical Outcomes · doi:10.1093/ehjqcco/qcad017 · Cited by 132
Ma, J., Dhiman, P., Qi, C., Bullock, G. S., van Smeden, M., Riley, R. D., & Collins, P. G. S. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.07.017 · Cited by 84
“[61,62].”
Self-citationMachine learning in predicting antimicrobial resistance: a systematic review and meta-analysis
Tang, R., Luo, R., Tang, S., Song, H., & Chen, X. (2022) · International Journal of Antimicrobial Agents · doi:10.1016/j.ijantimicag.2022.106684 · Cited by 93
Miles, J., Turner, J., Jacques, R., Williams, J., & Mason, S. M. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00084-1 · Cited by 91
Haines, K. J., Hibbert, E., McPeake, J. M., Anderson, B. J., Bienvenu, O. J., Andrews, A., Brummel, N. E., Ferrante, L. E., Hopkins, R. O., Hough, C. L., et al. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004659 · Cited by 80
Deception detection with machine learning: A systematic review and statistical analysis
Constâncio, A. S., Tsunoda, D. F., Silva, H. D. F. N., da Silveira, J. M., & Carvalho, D. R. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281323 · Cited by 102
Berkhout, W. E. M., van Wijngaarden, J. J., Workum, J. D., van de Sande, D., Hilling, D. E., Jung, C., Meyfroidt, G., Gommers, D., Buijsman, S. N. R., & van Genderen, M. E. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.22866 · Cited by 75
Artificial intelligence in salivary biomarker discovery and validation for oral diseases
Adeoye, J., & Su, Y. (2023) · Oral Diseases · doi:10.1111/odi.14641 · Cited by 86
de Jong, Y., Ramspek, C. L., Zoccali, C., Jager, K. J., Dekker, F. W., & van Diepen, M. (2021) · Nephrology · doi:10.1111/nep.13913 · Cited by 77
Vieira, S., Liang, X., Guiomar, R. N. R. M., & Mechelli, A. (2022) · Clinical Psychology Review · doi:10.1016/j.cpr.2022.102193 · Cited by 87
Bektaş, M., Tuynman, J. B., Pereira, J. C., Burchell, G. L., & van der Peet, D. L. (2022) · World Journal of Surgery · doi:10.1007/s00268-022-06728-1 · Cited by 83
Predicting relapse or recurrence of depression: systematic review of prognostic models
Moriarty, A. S., Meader, N., Snell, K. I. E., Riley, R. D., Paton, L. W., Dawson, S., Hendon, J., Chew-Graham, C. A., Gilbody, S., Churchill, R., et al. (2022) · The British Journal of Psychiatry · doi:10.1192/bjp.2021.218 · Cited by 71
Self-citationde Winter, M. A., Dorresteijn, J. A. N., Ageno, W., Ay, C., Beyer‐Westendorf, J., Coppens, M., Klok, F. A., Moustafa, F., Riva, N., Ruiz‐Artacho, P., et al. (2021) · Thrombosis and Haemostasis · doi:10.1055/s-0041-1735251 · Cited by 72
Chowdhury, M. Z. I., Naeem, I., Quan, H., Leung, A. A. C., Sikdar, K. C., O’Beirne, M., & Turin, T. C. (2022) · PLoS ONE · doi:10.1371/journal.pone.0266334 · Cited by 82
Brave New World of Artificial Intelligence: Its Use in Antimicrobial Stewardship—A Systematic Review
Pinto-de-Sá, R., Sousa‐Pinto, B., & Costa‐de‐Oliveira, S. (2024) · Antibiotics · doi:10.3390/antibiotics13040307 · Cited by 79
“To evaluate the risk of bias of the studies included in this review, the National Institute of Health (NIH) Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies and PROBAST (a tool to assess the risk of bias and applicability of prediction model studies) were used [48–50].”
MethodMortality prediction models in the adult critically ill: A scoping review
Keuning, B. E., Kaufmann, T., Wiersema, R., Granholm, A., Pettilä, V., Møller, M. H., Christiansen, C. F., Forte, J. C., Snieder, H., Keus, F., et al. (2019) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.13527 · Cited by 80
Heus, P., Reitsma, J. B., Collins, G. S., Damen, J. A. A. G., Scholten, R. J. P. M., Altman, D. G., Moons, K. G. M., & Hooft, L. (2020) · Annals of Internal Medicine · doi:10.7326/m20-0193 · Cited by 70
Self-citationvan Bussel, E. F., Hoevenaar‐Blom, M. P., Poortvliet, R. K. E., Gussekloo, J., van Dalen, J. W., van Gool, W. A., Richard, E., & van Charante, E. P. M. (2020) · Preventive Medicine · doi:10.1016/j.ypmed.2020.105986 · Cited by 88
Ogonah, M. G. T., Seyedsalehi, A., Whiting, D., & Fazel, S. (2023) · The Lancet Psychiatry · doi:10.1016/s2215-0366(23)00256-0 · Cited by 105
Wu, L., Lou, X., Kong, N., Xu, M., & Gao, C. (2022) · European Radiology · doi:10.1007/s00330-022-09174-8 · Cited by 76
“According to the PROBAST, all of the studies were considered to have a high ROB overall.”
MethodInfluential
Metadata
- Authors
- Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S.
- Affiliations
- Utrecht University (NL)
- Kleijnen Systematic Reviews (United Kingdom) (GB)
- Keele University (GB)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom (G.S.C.)
- Maastricht University (NL)
- Institute of Applied Health Research, National Institute for Health Research Birmingham Biomedical Research Centre, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom (S.M.)
- ORCID
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2019-01-01
- Volume
- 170
- Issue
- 1
- Pages
- W1-W33
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W2907638671
- Cited by
- 1,669
- References
- 201
- Citations by year
- 2026: 359 · 2025: 330 · 2024: 253 · 2023: 236 · 2022: 193 · 2021: 175 · 2020: 96 · 2019: 24
- FWCI
- 56.12
- Type
- journal article
- Language
- en
- Open access
- Licence
- other-oa
- www.acpjournals.org/journal/aim/text-and-data-mining (tdm)
- Locations
- Annals of Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- Utrecht University Repository (Utrecht University) (publishedVersion, other-oa)
- University of Birmingham Research Portal (University of Birmingham) (publishedVersion)
- Bristol Research (University of Bristol) (publishedVersion, other-oa)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Explore Bristol Research (submittedVersion, other-oa)
- Research Publications (Maastricht University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Funders
- National Institute for Health Research Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (ZONMW 918.10.615, ZONMW 912.08.004)
- National Institute for Health Research School for Primary Care Research (390)
- National Institute for Health Research Biomedical Research Centre, Oxford
- National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) West at University Hospitals Bristol NHS Foundation Trust
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- PROBAST, risk of bias assessment, applicability, prediction model, diagnostic model, prognostic model, model validation
- MeSH terms
- Systematic Reviews as Topic, Decision Support Techniques, Diagnosis, Humans, Prognosis, Research Design, Models, Statistical, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.