The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index; 6 of the 38 sources its claims cite were checked for support.
PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods
Open accessAbstract
The Prediction model Risk Of Bias ASsessment Tool (PROBAST) is used to assess the quality, risk of bias, and applicability of prediction models or algorithms and of prediction model/algorithm studies. Since PROBAST’s introduction in 2019, much progress has been made in the methodology for prediction modelling and in the use of artificial intelligence, including machine learning, techniques. An update to PROBAST-2019 is thus needed. This article describes the development of PROBAST+AI. PROBAST+AI consists of two distinctive parts: model development and model evaluation. For model development, PROBAST+AI users assess quality and applicability using 16 targeted signalling questions. For model evaluation, PROBAST+AI users assess the risk of bias and applicability using 18 targeted signalling questions. Both parts contain four domains: participants and data sources, predictors, outcome, and analysis. Applicability of the prediction model is rated for the participants and data sources, predictors, and outcome domains. PROBAST+AI may replace the original PROBAST tool and allows all key stakeholders (eg, model developers, AI companies, researchers, editors, reviewers, healthcare professionals, guideline developers, and policy organisations) to examine the quality, risk of bias, and applicability of any type of prediction model in the healthcare sector, irrespective of whether regression modelling or AI techniques are used.
Overview
CoChat Summary
## TL;DR
PROBAST+AI is an updated tool for judging the quality, risk of bias, and applicability of healthcare prediction models, expanding PROBAST-2019 to cover both regression and AI/machine learning methods. It separates model development from model evaluation and adds more explicit attention to fairness and algorithmic bias.
## Key findings
- PROBAST+AI contains two distinctive parts: model development and model evaluation; the development part uses 16 targeted signalling questions, and the evaluation part uses 18 targeted signalling questions.
- The tool has four domains: participants and data sources, predictors, outcome, and analysis; applicability is rated for participants and data sources, predictors, and outcome.
- In the final tool, PROBAST+AI includes 34 signalling questions in total (16 for model development, 18 for model evaluation) and six applicability items (three each for model development and evaluation).
- The paper states that recent reviews found most AI/machine learning prediction model studies to be poorly reported and poorly conducted, with both development and evaluation rated at high risk of bias; it also cites problems such as small sample sizes, poor handling of missing data, failure to deal with model overfitting, and lack of adequate assessment of predictive performance.
## Methods
- The tool was updated through a comprehensive, phased, international consensus process with multi-stakeholders.
- Development started from PROBAST-2019, then used a survey of more than 50 authors of a covid-19 prediction-model living review, followed by a three-round Delphi process and a hybrid consensus meeting.
- Delphi participation was: 95/201 invited in round 1, 144/294 in round 2 (12 partial responses), and 131/299 in round 3; the consensus meeting had 26/29 invited participants attending.
## Limitations
- The paper does not report a formal empirical validation study of PROBAST+AI’s measurement properties in this text; it presents the development process and guidance for use.
- As stated in the paper, algorithmic bias and fairness issues may not be identified by exploratory data analysis alone, and the ultimate assessment occurs when the model is deployed in daily healthcare practice.
- Evident from the text: the Delphi process required a change in data-capture platform after round 1 and had partial responses in round 2, which may have affected consistency across rounds.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
6 Valid · 1 Not valid · 23 Not checked · 41 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Algorithmic bias and fairness related issues may not be identified by exploratory data analysis alone.”
§ PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods > Discussion
What the sources say
- Partly supportedcites [10] sourceExperimental — accuracy not yet measured
The Results section ('Algorithmic encoding of attributes leads to fairness gaps'; page unavailable) assesses bias through differences in model error rates across demographic groups, and the Discussion says fairness may not transfer under distribution shift. These findings support the need for model-specific fairness assessment, but the paper does not directly evaluate whether exploratory data analysis alone would identify such issues. (LLM, conf=0.74)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“PROBAST+AI may replace the original PROBAST tool and allows all key stakeholders (eg, model developers, AI companies, researchers, editors, reviewers, healthcare professionals, guideline developers, and policy organisations) to examine the quality, risk of bias, and applicability of any type of prediction model in the healthcare sector, irrespective of whether regression modelling or AI techniques”
§ PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods > Abstract
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Prediction model studies including AI/machine learning based prediction model studies, also systematically are beset by overinterpretation (otherwise known as spin) of the applicability, validity, and usefulness of the resulting models.”
§ PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods > Abstract
What the sources say
- Not matchedcites 11 12
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“reported predictive performances are at high risk of bias”
§ PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods > Abstract
What the sources say
- Not matchedcites 2 3 4
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“fairness related issues affect the predictive performance of models related to certain groups.”
§ PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods > Abstract
What the sources say
- Not matchedcites 9 10
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
12 more claims
Create a free account to see all 71 claims and every source verdict.
- It may state two separate claims as one. 22
- Probably not a substantial claim. 44
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent guidance now makes clear that clinical prediction studies using machine learning should report not only discrimination, but also calibration, intended use, validation strategy, and risk-of-bias considerations in a structured way (9, 10).”
From AI-driven cardiovascular risk prediction in patients with diabetes: bridging algorithmic innovation to equitable clinical application (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“PROBAST-AI framework complements this by offering a structured tool to assess risk of bias and applicability in prediction model studies [69].”
From Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22. (2025) · cites this paper as [69]
The cited source says what the claim attributes to it.
In the citing paper
“Recent methodological frameworks, including TRIPOD+AI and PROBAST+AI, further support this shift from performance-centered development toward clinically trustworthy and equitable prediction modeling (9, 10).”
From AI-driven cardiovascular risk prediction in patients with diabetes: bridging algorithmic innovation to equitable clinical application (2026) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“At the same time, adherence to TRIPOD+AI and PROBAST+AI should be treated as a methodological expectation, not an optional add-on, because transparency and structured bias appraisal are necessary for evaluating whether apparently strong models are actually fit for clinical use (9, 10).”
From AI-driven cardiovascular risk prediction in patients with diabetes: bridging algorithmic innovation to equitable clinical application (2026) · cites this paper as [10]
The cited source says what the claim attributes to it.
In the citing paper
“Reporting, evaluation, and appraisal frameworks including SPIRIT-AI, CONSORT-AI, DECIDE-AI, TRIPOD+AI, and PROBAST+AI have substantially strengthened the conceptual and reporting architecture for trustworthy clinical AI [10,11,12,13,14].”
From Beyond Model Development in Healthcare AI: Post-Development Robustness, Post-Deployment Monitoring, and Lifecycle Governance—A Scoping Review of Reviews (2026) · cites this paper as [13]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718 · cited 3×
“38 62 Fairness (see box 1) means that prediction models should be designed and used to avoid adverse discrimination against any group of individuals and not to perpetuate any inequities in healthcare provision and outcomes for patients or the general population.”
Self-citationWynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Self-citationPROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group† (2019) · Annals of Internal Medicine · doi:10.7326/m18-1376 · Cited by 2,984
Self-citationCalculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917
Self-citationCalibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650
Self-citationChristodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119
Moons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018 · cited 1×
“…intended use of the prediction model) criteria as provided by the guidance of the Cochrane Prognosis Methods group (https://methods.cochrane.org/ prognosis/) and described in CHARMS (checklist for critical appraisal and data extraction for systematic reviews of prediction modelling studies 61 ).”
Self-citationMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Self-citationRiley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409
Self-citationPrediction models for cardiovascular disease risk in the general population: systematic review
Damen, J. A. A. G., Hooft, L., Schuit, E., Debray, T. P. A., Collins, G. S., Tzoulaki, I., Lassale, C. M., Siontis, G. C. M., Chiocchia, V., Roberts, C., et al. (2016) · BMJ · doi:10.1136/bmj.i2416 · Cited by 935
Self-citation
Research works printed but not linked: 1
- 10.1177/962280218784726 · doi:10.1177/962280218784726
Cited by
Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22.
Cecconi, M., Greco, M., Shickel, B., Angus, D. C., Bailey, H., Bignami, E., Calandra, T., Celi, L. A., Einav, S., Elbers, P., et al. (2025) · Critical Care · doi:10.1186/s13054-025-05532-2 · Cited by 80
“PROBAST-AI framework complements this by offering a structured tool to assess risk of bias and applicability in prediction model studies [69].”
Self-citationArtificial Intelligence in Dentistry: A Concise Review of Reporting Checklists and Guidelines
Khurshid, Z., Osathanon, T., Shire, M. A., Schwendicke, F., & Samaranayake, L. P. (2025) · International Dental Journal · doi:10.1016/j.identj.2025.109322 · Cited by 53
Liu, F., Beck, S., Yang, L., Luo, H., & Zhang, K. C. (2026) · Nature reviews. Cancer · doi:10.1038/s41568-026-00922-2 · Cited by 67
Artificial intelligence tool development: what clinicians need to know?
Chew, B.-H., & Ngiam, K. Y. (2025) · BMC Medicine · doi:10.1186/s12916-025-04076-0 · Cited by 41
The ethics of data mining in healthcare: challenges, frameworks, and future directions
Ahmed, M. M., Okesanya, O. J., Oweidat, M. H., Othman, Z. K., Musa, S. S., & Lucero‐Prisno, D. E. (2025) · BioData Mining · doi:10.1186/s13040-025-00461-w · Cited by 54
Metadata
- Authors
- Moons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., Denniston, A. K., Ghassemi, M., Heinze, G., Kengne, A. P., Maier-Hein, L., Liu, X., Logullo, P., McCradden, M. D., Liu, N., Oakden-Rayner, L., Singh, K., Ting, D. S., Wynants, L., Yang, B., Reitsma, J. B., Riley, R. D., Collins, G. S., & van Smeden, M.
- Affiliations
- Utrecht University (NL)
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, 3508 GA Utrecht, Netherlands
- University of Oxford (GB)
- Harvard University (US)
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- British Heart Foundation (GB)
- University of Birmingham (GB)
- Massachusetts Institute of Technology (US)
- Medical University of Vienna (AT)
- University of Cape Town (ZA)
- Division of Intelligent Medical Systems, German Cancer Research Centre (DKFZ), Heidelberg, Germany
- University Hospitals Birmingham NHS Foundation Trust (GB)
- Centre for Statistics in Medicine, UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
- Hospital for Sick Children (CA)
- Duke-NUS Medical School (SG)
- Australian Institute for Machine Learning, University of Adelaide, Adelaide, SA, Australia
- University of Michigan (US)
- National University of Singapore (SG)
- Maastricht University (NL)
- NIHR Birmingham Biomedical Research Centre (GB)
- Corresponding author
- Moons, Karel G M
- ORCID
- Moons, Karel G M
- Damen, Johanna A A
- Kaul, Tabea
- Hooft, Lotty
- Andaur Navarro, Constanza
- Dhiman, Paula
- Beam, Andrew L
- Van Calster, Ben
- Celi, Leo Anthony
- Denaxas, Spiros
- Denniston, Alastair K
- Ghassemi, Marzyeh
- Heinze, Georg
- Kengne, André Pascal
- Maier-Hein, Lena
- Liu, Xiaoxuan
- Logullo, Patricia
- McCradden, Melissa D
- Liu, Nan
- Oakden-Rayner, Lauren
- Singh, Karandeep
- Ting, Daniel S
- Wynants, Laure
- Yang, Bada
- Reitsma, Johannes B
- Riley, Richard D
- Collins, Gary S
- van Smeden, Maarten
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2025-03-24
- Volume
- 388
- Pages
- e082505
- ISSN
- 1756-1833 (ISSN-L)
- PMC
- PMC11931409
- OpenAlex
- W4408806526
- Cited by
- 1,069
- References
- 60
- Citations by year
- 2026: 957 · 2025: 112
- FWCI
- 677.97
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- BMJ (publishedVersion, cc-by-nc)
- Research Publications (Maastricht University) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Pure Amsterdam UMC (publishedVersion, other-oa)
- University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by-nc)
- Lirias (submittedVersion, cc-by-nc)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Funders
- Engineering and Physical Sciences Research Council (EP/Y018516/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- PROBAST+AI, PROBAST, risk of bias, applicability assessment, machine learning, regression models, prediction model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. https://app.cochat.ai/papers/10.1136/bmj-2024-082505
- The paper
- Moons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., Denniston, A. K., Ghassemi, M., Heinze, G., Kengne, A. P., Maier-Hein, L., Liu, X., Logullo, P., McCradden, M. D., Liu, N., . . . van Smeden, M. (2025). PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ. https://doi.org/10.1136/bmj-2024-082505
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.