PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Clinical prediction models combine multiple predictors to estimate risk for the presence of a particular condition (diagnostic models) or the occurrence of a certain event in the future (prognostic models). PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool for assessing the risk of bias (ROB) and applicability of diagnostic and prognostic prediction model studies, was developed by a steering group that considered existing ROB tools and reporting guidelines. The tool was informed by a Delphi procedure involving 38 experts and was refined through piloting. PROBAST is organized into the following 4 domains: participants, predictors, outcome, and analysis. These domains contain a total of 20 signaling questions to facilitate structured judgment of ROB, which was defined to occur when shortcomings in study design, conduct, or analysis lead to systematically distorted estimates of model predictive performance. PROBAST enables a focused and transparent approach to assessing the ROB and applicability of studies that develop, validate, or update prediction models for individualized predictions. Although PROBAST was designed for systematic reviews, it can be used more generally in critical appraisal of prediction model studies. Potential users include organizations supporting decision making, researchers and clinicians who are interested in evidence-based medicine or involved in guideline development, journal editors, and manuscript reviewers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329 · cited 3×
“A domain-based structure was adopted similar to that used in other risk of bias tools such as the revised Cochrane Risk of Bias tool,(29) QUADAS-2,(36) ROBINSI(30) and ROBIS.”
BackgroundMethodSelf-citationQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199 · cited 3×
“A domain-based structure was adopted similar to that used in other risk of bias tools such as the revised Cochrane Risk of Bias tool,(29) QUADAS-2,(36) ROBINSI(30) and ROBIS.”
InfluentialSelf-citationCochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,178
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · cited 11×
“A multivariable prediction model is defined as any combination or equation of two or more predictors for estimating the probability or risk for an individual.(7, 8, 33-35)”
InfluentialSelf-citation2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.005 · Cited by 4,945
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Circulation · doi:10.1161/01.cir.0000437741.48606.98 · Cited by 3,697
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Self-citationAssessing Bias in Studies of Prognostic Factors
Hayden, J. A., van der Windt, D. A., Cartwright, J. L., Côté, P., & Bombardier, C. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-4-201302190-00009 · Cited by 3,287 · cited 1×
“(25, 26) The QUIPS tool has been developed to assess the risk of bias in predictor finding (prognostic factor) studies.(28) The methodological quality of studies investigating the impact of a prediction model using a comparative randomised design can be assessed using the revised Cochrane risk of bias tool (ROB 2.”
MethodCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Self-citationSystematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
Clinical Practice Guidelines We Can Trust
Committee on Standards for Developing Trustworthy Clinical Practice Guidelines, Board on Health Care Services, & Institute of Medicine (2011) · National Academies Press eBooks · doi:10.17226/13058 · Cited by 3,081 · cited 1×
“Systematic reviews have a key role in evidence based medicine and in the development of clinical guidelines.(22-24) They are considered to provide the most reliable form of evidence for the effects of an intervention or diagnostic test.”
BackgroundROBIS: A new tool to assess risk of bias in systematic reviews was developed
Whiting, P., Savović, J., Higgins, J. P. T., Caldwell, D. M., Reeves, B. C., Shea, B., Davies, P., Kleijnen, J., & Churchill, R. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.06.005 · Cited by 2,272
Self-citationMoons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018 · cited 6×
“Assessors are first asked to report their systematic review question based on the guidance given in the CHARMS checklist.(41)”
InfluentialSelf-citationRisk Prediction Models for Hospital Readmission: A Systematic Review
Kansagara, D., Englander, H., Salanitro, A., Kagen, D., Theobald, C., Freeman, M., & Kripalani, S. (2011) · JAMA · doi:10.1001/jama.2011.1515 · Cited by 1,707 · cited 1×
“Prediction models, both diagnostic and prognostic, are widely used for a variety of medical domains and settings,(14-16) evidenced by the large number of models developed, especially in cancer,(17, 18) neurology,(19, 20) and cardiovascular disease domains.”
BackgroundMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Self-citationPrognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508
Self-citationPredicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2
Hippisley-Cox, J., Coupland, C., Vinogradova, Y., Robson, J., Minhas, R., Sheikh, A., & Brindle, P. (2008) · BMJ · doi:10.1136/bmj.39609.449676.25 · Cited by 1,459
Prediction of Postoperative Pulmonary Complications in a Population-based Surgical Cohort
Canet, J., Gallart, L., Gomar, C., Paluzie, G., Vallès, J., Castillo, J., Sabaté, S., Mazo, V., Briones, Z., & Sanchis, J. (2010) · Anesthesiology · doi:10.1097/aln.0b013e3181fc6e0a · Cited by 1,459 · cited 2×
“A multivariable prediction model is defined as any combination or equation of two or more predictors for estimating the probability or risk for an individual.(7, 8, 33-35)”
BackgroundGuidance for Developers of Health Research Reporting Guidelines
Moher, D., Schulz, K. F., Simera, I., & Altman, D. G. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000217 · Cited by 1,437 · cited 1×
“Development of PROBAST was based on a four-stage approach for developing health research reporting guidelines: define the scope, review the evidence base, web-based Delphi procedure, and refine the tool through piloting.(31) Guidelines explicitly aimed at the development of quality assessment tools were not available…”
Prognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436 · cited 5×
“Round 6 refined the domains and introduced further optional guidance for the use of PROBAST.”
InfluentialSelf-citationPrognosis and prognostic research: what, why, and how?
Moons, K. G. M., Royston, P., Vergouwe, Y., Grobbee, D. E., & Altman, D. G. (2009) · BMJ · doi:10.1136/bmj.b375 · Cited by 1,337 · cited 7×
“A multivariable prediction model is defined as any combination or equation of two or more predictors for estimating the probability or risk for an individual.(7, 8, 33-35)”
InfluentialSelf-citationPrognosis and prognostic research: Developing a prognostic model
Royston, P., Moons, K. G. M., Altman, D. G., & Vergouwe, Y. (2009) · BMJ · doi:10.1136/bmj.b604 · Cited by 1,172 · cited 5×
“Round 6 refined the domains and introduced further optional guidance for the use of PROBAST.”
InfluentialSelf-citationPrediction models for cardiovascular disease risk in the general population: systematic review
Damen, J. A. A. G., Hooft, L., Schuit, E., Debray, T. P. A., Collins, G. S., Tzoulaki, I., Lassale, C. M., Siontis, G. C. M., Chiocchia, V., Roberts, C., et al. (2016) · BMJ · doi:10.1136/bmj.i2416 · Cited by 935 · cited 1×
“Prediction models, both diagnostic and prognostic, are widely used for a variety of medical domains and settings,(14-16) evidenced by the large number of models developed, especially in cancer,(17, 18) neurology,(19, 20) and cardiovascular disease domains.(21) Prediction models are sometimes described as risk predicti…”
BackgroundSelf-citationPrognosis and prognostic research: application and impact of prognostic models in clinical practice
Moons, K. G. M., Altman, D. G., Vergouwe, Y., & Royston, P. (2009) · BMJ · doi:10.1136/bmj.b606 · Cited by 934 · cited 5×
“Round 6 refined the domains and introduced further optional guidance for the use of PROBAST.”
InfluentialSelf-citationPrognosis Research Strategy (PROGRESS) 2: Prognostic Factor Research
Riley, R. D., Hayden, J. A., Steyerberg, E. W., Moons, K. G. M., Abrams, K., Kyzas, P. A., Malats, N., Briggs, A., Schroter, S., Altman, D. G., et al. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001380 · Cited by 823 · cited 7×
“extend) a model for making individualised predictions are not covered by PROBAST.(6) Hence PROBAST is not intended for predictor finding studies and prediction model impact studies.”
InfluentialSelf-citationPrognosis research strategy (PROGRESS) 1: A framework for researching clinical outcomes
Hemingway, H., Croft, P., Perel, P., Hayden, J. A., Abrams, K., Timmis, A., Briggs, A., Udumyan, R., Moons, K. G. M., Steyerberg, E. W., et al. (2013) · BMJ · doi:10.1136/bmj.e5595 · Cited by 663 · cited 5×
“Round 6 refined the domains and introduced further optional guidance for the use of PROBAST.”
InfluentialSelf-citationA guide to systematic review and meta-analysis of prediction model performance
Debray, T. P. A., Damen, J. A. A. G., Snell, K. I. E., Ensor, J., Hooft, L., Reitsma, J. B., Riley, R. D., & Moons, K. G. M. (2017) · BMJ · doi:10.1136/bmj.i6460 · Cited by 657 · cited 3×
“Systematic reviews of prediction model studies are a rapidly evolving area.(27) PROBAST is the first rigorously developed tool designed specifically to assess the quality of prediction model studies for development, validation or updating models for both diagnostic and prognostic models.”
Self-citationReporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600 · cited 13×
“(1) Predictor finding studies (also known as risk factor or prognostic factor studies) aim to identify which predictors independently contribute to the prediction of a diagnostic or prognostic outcome.(1, 6) Prediction model studies typically aim to develop, validate or adjust (e.”
InfluentialSelf-citationCollins, G. S., Mallett, S., Omar, O., & Yu, L.-M. (2011) · BMC Medicine · doi:10.1186/1741-7015-9-103 · Cited by 484 · cited 1×
“Prediction models, both diagnostic and prognostic, are widely used for a variety of medical domains and settings,(14-16) evidenced by the large number of models developed, especially in cancer,(17, 18) neurology,(19, 20) and cardiovascular disease domains.”
BackgroundSelf-citationShariat, S. F., Karakiewicz, P. I., Suardi, N., & Kattan, M. W. (2008) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-07-4713 · Cited by 288 · cited 1×
“Prediction models, both diagnostic and prognostic, are widely used for a variety of medical domains and settings,(14-16) evidenced by the large number of models developed, especially in cancer,(17, 18) neurology,(19, 20) and cardiovascular disease domains.”
Systematic Review of Prognostic Models in Patients with Acute Stroke
Counsell, C., & Dennis, M. (2001) · Cerebrovascular Diseases · doi:10.1159/000047699 · Cited by 250 · cited 1×
“Prediction models, both diagnostic and prognostic, are widely used for a variety of medical domains and settings,(14-16) evidenced by the large number of models developed, especially in cancer,(17, 18) neurology,(19, 20) and cardiovascular disease domains.”
BackgroundReporting methods in studies developing prognostic models in cancer: a review
Mallett, S., Royston, P., Dutton, S., Waters, R., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-20 · Cited by 206 · cited 5×
“Round 6 refined the domains and introduced further optional guidance for the use of PROBAST.”
InfluentialSelf-citationCollins, G. S., Omar, O., Shanyinde, M., & Yu, L.-M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.06.020 · Cited by 192 · cited 7×
“A multivariable prediction model is defined as any combination or equation of two or more predictors for estimating the probability or risk for an individual.(7, 8, 33-35)”
InfluentialThe Wells Rule Does Not Adequately Rule Out Deep Venous Thrombosis in Primary Care Patients
Oudega, R., Hoes, A. W., & Moons, K. G. M. (2005) · Annals of Internal Medicine · doi:10.7326/0003-4819-143-2-200507190-00008 · Cited by 148 · cited 2×
“Well known examples are the Wells rule for diagnosing deep venous thrombosis,(9) the Ottawa ankle rules for detecting fractures,(10, 11) QRISK2 for predicting cardiovascular risk,(12) and the PCPT risk calculator for prostate biopsies.”
Self-citationPrognostic Models: A Methodological Framework and Review of Models for Breast Cancer
Altman, D. G. (2009) · Cancer Investigation · doi:10.1080/07357900802572110 · Cited by 142 · cited 1×
“Prediction models, both diagnostic and prognostic, are widely used for a variety of medical domains and settings,(14-16) evidenced by the large number of models developed, especially in cancer,(17, 18) neurology,(19, 20) and cardiovascular disease domains.”
BackgroundLamain-de Ruiter, M., Kwee, A., Naaktgeboren, C. A., de Groot, I., Evers, I. M., Groenendaal, F., Hering, Y. R., Huisjes, A. J. M., Kirpestein, C., Monincx, W. M., et al. (2016) · BMJ · doi:10.1136/bmj.i4338 · Cited by 123 · cited 2×
“(1-3) Prognosis does not only pertain to sick individuals or with an established diagnosis, but also to, for example, prognosis of pregnant women at risk of developing diabetes(4) or of individuals in the general population at risk of developing osteoporotic fractures(5).”
BackgroundSelf-citationA proposed framework for developing quality assessment tools
Whiting, P., Wolff, R., Mallett, S., Simera, I., & Savović, J. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0604-6 · Cited by 121 · cited 1×
“(31) Guidelines explicitly aimed at the development of quality assessment tools were not available at the time.(32)”
BackgroundSelf-citationPerel, P., Prieto‐Merino, D., Shakur‐Still, H., Clayton, T. C., Lecky, F. E., Bouamra, O., Russell, R. J., Faulkner, M., Steyerberg, E. W., & Roberts, I. (2012) · BMJ · doi:10.1136/bmj.e5166 · Cited by 103
Risk assessment of fragility fractures: summary of NICE guidance
Rabar, S., Lau, R., O'Flynn, N., Li, L., Barry, P., & on behalf of the Guideline Development Group (2012) · BMJ · doi:10.1136/bmj.e3698 · Cited by 74 · cited 1×
“Systematic reviews have a key role in evidence based medicine and in the development of clinical guidelines.(22-24) They are considered to provide the most reliable form of evidence for the effects of an intervention or diagnostic test.”
BackgroundBetween iatrotropic stimulus and interiatric referral:
Knottnerus, J. A. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(02)00528-0 · Cited by 62
DIAGNOSTIC PREDICTION RULES: PRINCIPLES, REQUIREMENTS, AND PITFALLS
Knottnerus, J. A. (1995) · Primary Care: Clinics in Office Practice · doi:10.1016/s0095-4543(21)00522-4 · Cited by 56
Steurer, J., Haller, C., Häuselmann, H., Brunner, F., & Bachmann, L. M. (2011) · PLoS ONE · doi:10.1371/journal.pone.0019994 · Cited by 25 · cited 1×
“Prediction models, both diagnostic and prognostic, are widely used for a variety of medical domains and settings,(14-16) evidenced by the large number of models developed, especially in cancer,(17, 18) neurology,(19, 20) and cardiovascular disease domains.”
Background
Cited by
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
Self-citationCalculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917
Self-citationSCORE2 working group and ESC Cardiovascular risk collaboration, Hageman, S., Pennells, L., Ojeda, F., Kaptoge, S., Kuulasmaa, K., de Vries, T., Xu, Z., Kee, F., Chung, R., et al. (2021) · European Heart Journal · doi:10.1093/eurheartj/ehab309 · Cited by 1,810
Self-citationMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Self-citationMa, L.-L., Wang, Y.-Y., Yang, Z.-H., Huang, D., Weng, H., & Zeng, X.-T. (2020) · Military Medical Research · doi:10.1186/s40779-020-00238-8 · Cited by 1,648
Metadata
- Authors
- Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group†
- Affiliations
- Kleijnen Systematic Reviews (United Kingdom) (GB)
- Utrecht University (NL)
- Keele University (GB)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom (G.S.C.)
- Maastricht University (NL)
- Institute of Applied Health Research, National Institute for Health Research Birmingham Biomedical Research Centre, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom (S.M.)
- ORCID
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2019-01-01
- Volume
- 170
- Issue
- 1
- Pages
- 51-58
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W2907554860
- Cited by
- 2,984
- References
- 42
- Citations by year
- 2026: 921 · 2025: 561 · 2024: 416 · 2023: 343 · 2022: 303 · 2021: 250 · 2020: 140 · 2019: 45
- FWCI
- 83.39
- Type
- journal article
- Language
- en
- Open access
- Licence
- other-oa
- www.acpjournals.org/journal/aim/text-and-data-mining (tdm)
- Locations
- Annals of Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- Bristol Research (University of Bristol) (acceptedVersion, other-oa)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- FreiDok plus (Universitätsbibliothek Freiburg) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- Radboud Repository (Radboud University) (submittedVersion)
- Explore Bristol Research (submittedVersion, other-oa)
- Apollo (University of Cambridge) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Apollo
- Funders
- National Institute for Health Research Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (ZONMW 918.10.615, ZONMW 912.08.004)
- National Institute for Health Research School for Primary Care Research (390)
- National Institute for Health Research Biomedical Research Centre, Oxford
- National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) West at University Hospitals Bristol NHS Foundation Trust
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- PROBAST, risk of bias assessment, diagnostic model, prognostic model, applicability, critical appraisal, reporting guidelines, Delphi consensus, systematic reviews, personalized prediction
- MeSH terms
- Systematic Reviews as Topic, Decision Support Techniques, Delphi Technique, Diagnosis, Humans, Prognosis, Research Design, Models, Statistical, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.