Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index; 6 of the 38 sources its claims cite were checked for support.

PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods

Open access
Moons, Karel G M, Damen, Johanna A A, Kaul, Tabea, Hooft, Lotty, Andaur Navarro, Constanza, Dhiman, Paula, Beam, Andrew L, Van Calster, Ben, Celi, Leo Anthony, Denaxas, Spiros, Denniston, Alastair K, Ghassemi, Marzyeh, Heinze, Georg, Kengne, André Pascal, Maier-Hein, Lena, Liu, Xiaoxuan, Logullo, Patricia, McCradden, Melissa D, Liu, Nan, Oakden-Rayner, Lauren, Singh, Karandeep, Ting, Daniel S, Wynants, Laure, Yang, Bada, Reitsma, Johannes B, Riley, Richard D, Collins, Gary S, van Smeden, Maarten · 2025 · BMJ

Abstract

The Prediction model Risk Of Bias ASsessment Tool (PROBAST) is used to assess the quality, risk of bias, and applicability of prediction models or algorithms and of prediction model/algorithm studies. Since PROBAST’s introduction in 2019, much progress has been made in the methodology for prediction modelling and in the use of artificial intelligence, including machine learning, techniques. An update to PROBAST-2019 is thus needed. This article describes the development of PROBAST+AI. PROBAST+AI consists of two distinctive parts: model development and model evaluation. For model development, PROBAST+AI users assess quality and applicability using 16 targeted signalling questions. For model evaluation, PROBAST+AI users assess the risk of bias and applicability using 18 targeted signalling questions. Both parts contain four domains: participants and data sources, predictors, outcome, and analysis. Applicability of the prediction model is rated for the participants and data sources, predictors, and outcome domains. PROBAST+AI may replace the original PROBAST tool and allows all key stakeholders (eg, model developers, AI companies, researchers, editors, reviewers, healthcare professionals, guideline developers, and policy organisations) to examine the quality, risk of bias, and applicability of any type of prediction model in the healthcare sector, irrespective of whether regression modelling or AI techniques are used.

Overview

CoChat Summary

## TL;DR

PROBAST+AI is an updated tool for judging the quality, risk of bias, and applicability of healthcare prediction models, expanding PROBAST-2019 to cover both regression and AI/machine learning methods. It separates model development from model evaluation and adds more explicit attention to fairness and algorithmic bias.

## Key findings

  • PROBAST+AI contains two distinctive parts: model development and model evaluation; the development part uses 16 targeted signalling questions, and the evaluation part uses 18 targeted signalling questions.
  • The tool has four domains: participants and data sources, predictors, outcome, and analysis; applicability is rated for participants and data sources, predictors, and outcome.
  • In the final tool, PROBAST+AI includes 34 signalling questions in total (16 for model development, 18 for model evaluation) and six applicability items (three each for model development and evaluation).
  • The paper states that recent reviews found most AI/machine learning prediction model studies to be poorly reported and poorly conducted, with both development and evaluation rated at high risk of bias; it also cites problems such as small sample sizes, poor handling of missing data, failure to deal with model overfitting, and lack of adequate assessment of predictive performance.

## Methods

  • The tool was updated through a comprehensive, phased, international consensus process with multi-stakeholders.
  • Development started from PROBAST-2019, then used a survey of more than 50 authors of a covid-19 prediction-model living review, followed by a three-round Delphi process and a hybrid consensus meeting.
  • Delphi participation was: 95/201 invited in round 1, 144/294 in round 2 (12 partial responses), and 131/299 in round 3; the consensus meeting had 26/29 invited participants attending.

## Limitations

  • The paper does not report a formal empirical validation study of PROBAST+AI’s measurement properties in this text; it presents the development process and guidance for use.
  • As stated in the paper, algorithmic bias and fairness issues may not be identified by exploratory data analysis alone, and the ultimate assessment occurs when the model is deployed in daily healthcare practice.
  • Evident from the text: the Delphi process required a change in data-capture platform after round 1 and had partial responses in round 2, which may have affected consistency across rounds.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1136/bmj-2024-082505

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

6 Valid · 1 Not valid · 23 Not checked · 41 Own results and interpretations

12 more claims

Create a free account to see all 71 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 22
  • Probably not a substantial claim. 44

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 60 references

Research works printed but not linked: 1

  1. 10.1177/962280218784726 · doi:10.1177/962280218784726

Cited by

See all 1,283

Metadata

Authors
Moons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., Denniston, A. K., Ghassemi, M., Heinze, G., Kengne, A. P., Maier-Hein, L., Liu, X., Logullo, P., McCradden, M. D., Liu, N., Oakden-Rayner, L., Singh, K., Ting, D. S., Wynants, L., Yang, B., Reitsma, J. B., Riley, R. D., Collins, G. S., & van Smeden, M.
Affiliations
  • Utrecht University (NL)
  • Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, 3508 GA Utrecht, Netherlands
  • University of Oxford (GB)
  • Harvard University (US)
  • Department of Development and Regeneration, KU Leuven, Leuven, Belgium
  • British Heart Foundation (GB)
  • University of Birmingham (GB)
  • Massachusetts Institute of Technology (US)
  • Medical University of Vienna (AT)
  • University of Cape Town (ZA)
  • Division of Intelligent Medical Systems, German Cancer Research Centre (DKFZ), Heidelberg, Germany
  • University Hospitals Birmingham NHS Foundation Trust (GB)
  • Centre for Statistics in Medicine, UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
  • Hospital for Sick Children (CA)
  • Duke-NUS Medical School (SG)
  • Australian Institute for Machine Learning, University of Adelaide, Adelaide, SA, Australia
  • University of Michigan (US)
  • National University of Singapore (SG)
  • Maastricht University (NL)
  • NIHR Birmingham Biomedical Research Centre (GB)
Corresponding author
Moons, Karel G M
Venue
BMJ
Publisher
BMJ
Published
2025-03-24
Volume
388
Pages
e082505
ISSN
1756-1833 (ISSN-L)
OpenAlex
W4408806526
Cited by
1,069
References
60
Citations by year
2026: 957 · 2025: 112
FWCI
677.97
Type
journal article
Language
en
Open access
Locations
  • BMJ (publishedVersion, cc-by-nc)
  • Research Publications (Maastricht University) (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • Pure Amsterdam UMC (publishedVersion, other-oa)
  • University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by-nc)
  • Lirias (submittedVersion, cc-by-nc)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
Funders
Engineering and Physical Sciences Research Council (EP/Y018516/1)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
PROBAST+AI, PROBAST, risk of bias, applicability assessment, machine learning, regression models, prediction model evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. https://app.cochat.ai/papers/10.1136/bmj-2024-082505
The paper
Moons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., Denniston, A. K., Ghassemi, M., Heinze, G., Kengne, A. P., Maier-Hein, L., Liu, X., Logullo, P., McCradden, M. D., Liu, N., . . . van Smeden, M. (2025). PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ. https://doi.org/10.1136/bmj-2024-082505

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.