Poor Reliability between Cochrane Reviewers and Blinded External Reviewers When Applying the Cochrane Risk of Bias Tool in Physical Therapy Trials
Open accessThe DOI matches its Crossref record; 60 of the 61 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: To test the inter-rater reliability of the RoB tool applied to Physical Therapy (PT) trials by comparing ratings from Cochrane review authors with those of blinded external reviewers. METHODS: Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Data on RoB assessments from Cochrane reviews and other characteristics of reviews and trials were extracted. Consensus assessments between the two reviewers were then compared with the RoB ratings from the Cochrane reviews. Agreement between Cochrane and blinded external reviewers was assessed using weighted kappa (κ). RESULTS: In total, 109 trials included in 17 Cochrane reviews were assessed. Inter-rater reliability on the overall RoB assessment between Cochrane review authors and blinded external reviewers was poor (κ = 0.02, 95%CI: -0.06, 0.06]). Inter-rater reliability on individual domains of the RoB tool was poor (median κ = 0.19), ranging from κ = -0.04 ("Other bias") to κ = 0.62 ("Sequence generation"). There was also no agreement (κ = -0.29, 95%CI: -0.81, 0.35]) in the overall RoB assessment at the meta-analysis level. CONCLUSIONS: Risk of bias assessments of RCTs using the RoB tool are not consistent across different research groups. Poor agreement was not only demonstrated at the trial level but also at the meta-analysis level. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to consistently apply the RoB tool and revisions to the tool for different health areas are needed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, RoB assessment has long been recognized as labor intensive, variably reproducible, and prone to misapplication in specific domains such as selective reporting [14-16].”
From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [15]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“however, we developed specific decision rules to make decisions as described elsewhere.43”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [43]
In the citing paper
“Risk of bias assessment requires judgement and numerous studies have reported low inter-rater reliability of the RoB1 tool [10, 11].”
From Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system (2022) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · cited 1×
“Statistical analysis Inter-rater reliability of RoB assessments between Cochrane and blinded external reviewers [18–20] and within the panel of external reviewers was assessed using weighted kappa (k) for categorical data.”
MethodSchulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634 · cited 6×
“Some studies, for example, have found that inadequate allocation concealment or lack of double-blinding can overestimate treatment effects on average by 18% and 9%, respectively [5,40,42].”
BackgroundMethodInfluential(2008) · Cited by 3,790
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · cited 7×
“The Cochrane Collaboration developed the Risk of Bias tool (RoB) as a method to assess risk of bias based on study design and conduct rather than relying on general reporting issues of trial characteristics [3].”
BackgroundMethodInfluentialWood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538 · cited 4×
“For example, authors of meta-epidemiological studies [8,39,40], have taken information from external sources (Cochrane assessments, or information provided by authors of reviews).”
BackgroundResultInfluentialSelf-citation2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group
Furlan, A. D., Pennick, V., Bombardier, C., & van Tulder, M. (2009) · Spine · doi:10.1097/brs.0b013e3181b1c99f · Cited by 1,663
Armijo‐Olivo, S., Stiles, C. R., Hagen, N. A., Biondo, P. D., & Cummings, G. G. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2010.01516.x · Cited by 1,539 · cited 5×
“Final ratings of the RoB tool based on the Cochrane reviewers assessments indicated that almost 92% of trials included in the reviews had either high or unclear RoB; a proportion that is similar to those identified in other studies [10,13].”
BackgroundMethodResultInfluentialSelf-citationMoher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)00031-w · Cited by 1,308 · cited 4×
“Different approaches to quality assessment have been proposed for assessing the quality of studies [1,2].”
BackgroundInfluentialEgger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181 · cited 1×
“For example, authors of meta-epidemiological studies [8,39,40], have taken information from external sources (Cochrane assessments, or information provided by authors of reviews).”
BackgroundSelf-citationSavović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049
Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease
Puhan, M. A., Gimeno-Santos, E., Cates, C. J., & Troosters, T. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005305.pub4 · Cited by 991
Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review
Olivo, S. A., Macedo, L. G., Gadotti, I. C., Fuentes, J., Stanton, T., & Magee, D. J. (2008) · Physical Therapy · doi:10.2522/ptj.20070147 · Cited by 829
Self-citationHome-based versus centre-based cardiac rehabilitation
Taylor, R. S., Dalal, H., Jolly, K., Moxham, T., & Zawada, A. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007130.pub2 · Cited by 623
Self-management education for patients with chronic obstructive pulmonary disease
Effing, T., Monninkhof, E. E., van der Valk, P. P. D. L. P. M., Zielhuis, G. G., Walters, E. H., van der Palen, J. J., & Zwerink, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002990.pub2 · Cited by 536 · cited 1×
“Characteristics of selected studies Briefly, the reviews were published between 2008 and 2011 and included meta-analyses of the effectiveness of PT interventions for musculoskeletal (9 reviews [22–30] cardiorespiratory (4 reviews) [31–34], neurological (2 reviews) [35,36], gynaecological (1 review) [37], and general c…”
BackgroundExercise for osteoarthritis of the hip
Fransen, M., McConnell, S., Hernandez-Molina, G., & Reichenbach, S. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007912.pub2 · Cited by 476
Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials
Pildal, J., Hrobjartsson, A., Jorgensen, K., Hilden, J., Altman, D., & Gotzsche, P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym087 · Cited by 472
Kjaergard, L., & Als-Nielsen, B. (2002) · BMJ · doi:10.1136/bmj.325.7358.249 · Cited by 433
Byrt, T. (1996) · Epidemiology · doi:10.1097/00001648-199609000-00030 · Cited by 390
Moher, Cook, Jadad, Tugwell, Moher, Jones, Pham, & Klassen (1999) · Health Technology Assessment · doi:10.3310/hta3120 · Cited by 368
Armijo-Olivo, S., Warren, S., & Magee, D. (2009) · Physical Therapy Reviews · doi:10.1179/174328809x405928 · Cited by 361 · cited 1×
“[43,44] Similarly, effect sizes from trials that excluded dropouts in the analysis or considered a modified intention to treat (ITT) approach were more likely to show a beneficial effect than trials without exclusions, demonstrating that the ITT principle is important to preserve the benefits of randomization and keep…”
BackgroundSelf-citationRecommendations by Cochrane Review Groups for assessment of the risk of bias in studies
Lundh, A., & Gøtzsche, P. C. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-22 · Cited by 352
Risk of bias versus quality assessment of randomised controlled trials: cross sectional study
Hartling, L., Ospina, M., Liang, Y., Dryden, D. M., Hooton, N., Krebs Seida, J., & Klassen, T. P. (2009) · BMJ · doi:10.1136/bmj.b4012 · Cited by 348 · cited 8×
“, 2009 [9] Graham et al.”
BackgroundMethodResultInfluentialSelf-citationExercise or exercise and diet for preventing type 2 diabetes mellitus
Orozco, L. J., Buchleitner, A. M., Gimenez-Perez, G., Roqué i Figuls, M., Richter, B., & Mauricio, D. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003054.pub3 · Cited by 327
Aerobic exercise for women during pregnancy
Kramer, M. S., & McDonald, S. W. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000180.pub2 · Cited by 324
Acupuncture for peripheral joint osteoarthritis
Manheimer, E., Cheng, K., Linde, K., Lao, L., Yoo, J., Wieland, S., van der Windt, D. A., Berman, B. M., & Bouter, L. M. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001977.pub2 · Cited by 302
Multidisciplinary rehabilitation for older people with hip fractures
Handoll, H. H., Cameron, I. D., Mak, J. C., & Finnegan, T. P. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007125.pub2 · Cited by 286
Promoting patient uptake and adherence in cardiac rehabilitation
Davies, P., Taylor, F., Beswick, A., Wise, F., Moxham, T., Rees, K., & Ebrahim, S. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007131.pub2 · Cited by 283
Transcutaneous electrostimulation for osteoarthritis of the knee
Rutjes, A. W., Nüesch, E., Sterchi, R., Kalichman, L., Hendriks, E., Osiri, M., Brosseau, L., Reichenbach, S., & Jüni, P. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002823.pub2 · Cited by 272
World Confederation for Physical Therapy
O'Hare, M. (1994) · Physiotherapy · doi:10.1016/s0031-9406(10)60887-5 · Cited by 260
Unnebrink, K., & Windeler, J. (2001) · Statistics in Medicine · doi:10.1002/sim.1149 · Cited by 258
Modified intention to treat reporting in randomised controlled trials: systematic review
Abraha, I., & Montedori, A. (2010) · BMJ · doi:10.1136/bmj.c2697 · Cited by 255
Nuesch, E., Trelle, S., Reichenbach, S., Rutjes, A. W. S., Burgi, E., Scherer, M., Altman, D. G., & Juni, P. (2009) · BMJ · doi:10.1136/bmj.b3244 · Cited by 248 · cited 1×
“[43,44] Similarly, effect sizes from trials that excluded dropouts in the analysis or considered a modified intention to treat (ITT) approach were more likely to show a beneficial effect than trials without exclusions, demonstrating that the ITT principle is important to preserve the benefits of randomization and keep…”
BackgroundEmerson, J. D., Burdick, E., Hoaglin, D. C., Mosteller, F., & Chalmers, T. C. (1990) · Controlled Clinical Trials · doi:10.1016/0197-2456(90)90175-2 · Cited by 241
Stretch for the treatment and prevention of contractures
Katalinic, O. M., Harvey, L. A., Herbert, R. D., Moseley, A. M., Lannin, N. A., & Schurr, K. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007455.pub2 · Cited by 228
Therapeutic ultrasound for osteoarthritis of the knee or hip
Rutjes, A. W., Nüesch, E., Sterchi, R., & Jüni, P. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003132.pub2 · Cited by 221
Rehabilitation after lumbar disc surgery
Oosterhuis, T., Costa, L. O., Maher, C. G., de Vet, H. C., van Tulder, M. W., & Ostelo, R. W. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003007.pub3 · Cited by 220
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207
Constraint-induced movement therapy for upper extremities in stroke patients
Sirtori, V., Corbetta, D., Moja, L., & Gatti, R. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004433.pub2 · Cited by 204
Hartling, L., Hamm, M. P., Milne, A., Vandermeer, B., Santaguida, P. L., Ansari, M., Tsertsvadze, A., Hempel, S., Shekelle, P., & Dryden, D. M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.07.005 · Cited by 173 · cited 10×
“Only one previous study assessed the reliability of the RoB based on consensus assessments across pairs of reviewers from four research centres using a sample of 30 trials indexed in PubMed between 2000 and 2006 [13].”
BackgroundMethodResultInfluentialIntention-to-treat approach to data from randomized controlled trials: a sensitivity analysis
Wright, C. C., & Sim, J. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00155-0 · Cited by 171 · cited 1×
“When the drop-out rate was higher than 20%, the item was scored as ‘‘high’’ risk of bias since there is evidence that drop-out rates higher than 20% are likely to increase bias in treatment estimates [52,53].”
BackgroundSchuck, P. (2004) · Quality of Life Research · doi:10.1023/b:qure.0000021318.92272.2a · Cited by 161
Continuous passive motion following total knee arthroplasty in people with arthritis
Harvey, L. A., Brosseau, L., & Herbert, R. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004260.pub3 · Cited by 145
Assessing the impact of attrition in randomized controlled trials
Hewitt, C. E., Kumaravel, B., Dumville, J. C., & Torgerson, D. J. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.01.010 · Cited by 139
Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease
Puhan, M. A., Gimeno-Santos, E., Scharplatz, M., Troosters, T., Walters, E. H., & Steurer, J. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005305.pub2 · Cited by 136
Continuous passive motion following total knee arthroplasty in people with arthritis
Harvey, L. A., Brosseau, L., & Herbert, R. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004260.pub2 · Cited by 131
Manzoli, L., Schioppa, F., Boccia, A., & Villari, P. (2007) · The Pediatric Infectious Disease Journal · doi:10.1097/01.inf.0000253053.01151.bd · Cited by 119
Overground physical therapy gait training for chronic stroke patients with mobility deficits
States, R. A., Pappas, E., & Salem, Y. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006075.pub2 · Cited by 110
Exercise for osteoarthritis of the hip
Fransen, M., McConnell, S., Hernandez-Molina, G., & Reichenbach, S. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007912 · Cited by 101
World Confederation for Physical Therapy
McKay, E. (1959) · Physical Therapy · doi:10.1093/ptj/39.1.36 · Cited by 100
Physical conditioning programs for improving work outcomes in workers with back pain
Schaafsma, F., Schonstein, E., Whelan, K. M., Ulvestad, E., Kenny, D. T., & Verbeek, J. H. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001822.pub2 · Cited by 92
Hartling, L., Bond, K., Vandermeer, B., Seida, J., Dryden, D. M., & Rowe, B. H. (2011) · PLoS ONE · doi:10.1371/journal.pone.0017242 · Cited by 78
Villari, P., Manzoli, L., & Boccia, A. (2004) · Vaccine · doi:10.1016/j.vaccine.2004.01.068 · Cited by 73
Trowman, R., Dumville, J. C., Torgerson, D. J., & Cranny, G. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.03.014 · Cited by 69
Rehabilitation after lumbar disc surgery
Ostelo, R., Costa, L. O. P., Maher, C. G., de Vet, H. C. W., & van Tulder, M. W. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003007.pub2 · Cited by 31
Graham, N., Haines, T., Goldsmith, C. H., Gross, A., Burnie, S., Shahzad, U., & Talovikova, E. (2012) · Spine · doi:10.1097/brs.0b013e31822671eb · Cited by 25
Classification systems to improve assessment of risk of bias
Boutron, I., & Ravaud, P. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.09.006 · Cited by 20
Cited by
Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2017) · doi:10.23970/ahrqepcmethguide2 · Cited by 448
Armijo-Olivo, S., Pitance, L., Singh, V., Neto, F., Thie, N., & Michelotti, A. (2016) · Physical Therapy · doi:10.2522/ptj.20140548 · Cited by 526
Self-citationJørgensen, L., Paludan-Müller, A. S., Laursen, D. R. T., Savović, J., Boutron, I., Sterne, J. A. C., Higgins, J. P. T., & Hróbjartsson, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0259-8 · Cited by 396
Savović, J., Turner, R. M., Mawdsley, D., Jones, H. E., Beynon, R., Higgins, J. P. T., & Sterne, J. A. C. (2018) · American Journal of Epidemiology · doi:10.1093/aje/kwx344 · Cited by 382
Schwingshackl, L., Knüppel, S., Schwedhelm, C., Hoffmann, G., Missbach, B., Stelmach-Mardas, M., Dietrich, S., Eichelmann, F., Kontopanteils, E., Iqbal, K., et al. (2016) · Advances in Nutrition · doi:10.3945/an.116.013052 · Cited by 360
Metadata
- Authors
- Armijo-Olivo, S., Ospina, M., da Costa, B. R., Egger, M., Saltaji, H., Fuentes, J., Ha, C., & Cummings, G. G.
- Affiliations
- University of Alberta (CA)
- Alberta Health Services (CA)
- University of Bern (CH)
- Corresponding author
- Armijo-Olivo, Susan
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-05-13
- Volume
- 9
- Issue
- 5
- Pages
- e96920
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4019638
- OpenAlex
- W2027833822
- Cited by
- 124
- References
- 61
- Citations by year
- 2026: 6 · 2025: 5 · 2024: 10 · 2023: 15 · 2022: 9 · 2021: 9 · 2020: 9 · 2019: 10 · 2018: 12 · 2017: 17 · 2016: 10 · 2015: 10 · 2014: 2
- FWCI
- 12.68
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Florida International University Digital Commons (Florida International University) (submittedVersion)
- Florida International University Digital Commons (Florida International University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- BORIS (University Library Bern) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Open Access CRIS of the University of Bern
- Funders
- Women and Children's Health Research Institute
- Physiotherapy Foundation of Canada
- Alberta Innovates - Health Solutions
- University of Alberta
- Fondation pour la Recherche Médicale
- Killam Trusts
- Alberta Innovates
- Canadian Institutes of Health Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- risk of bias assessment, Cochrane risk of bias tool, inter-rater reliability, consensus assessment
- MeSH terms
- Humans, Research Design, Review Literature as Topic, Reproducibility of Results, Bias, Randomized Controlled Trials as Topic, Physical Therapy Modalities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
