CoChat
Sign Up
Cited by 5Meta-analysis and systematic reviewsArtificial Intelligence in Healthcare and EducationStatistical Methods in Clinical Trials

Using a large language model (ChatGPT) to assess risk of bias in randomized controlled trials of medical interventions: protocol for a pilot study of interrater agreement with human reviewers

Open access
Rose, Christopher James, Bidonde, Julia, Ringsten, Martin, Glanville, Julie, Berg, Rigmor C., Cooper, Chris, Muller, Ashley Elizabeth, Bergsund, Hans Bugge +2 more·2025·BMC Medical Research Methodology
DOI 10.1186/s12874-025-02631-0PMC PMC12315198
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Risk of bias (RoB) assessment is an essential part of systematic reviews that requires reading and understanding each eligible trial and RoB tools. RoB assessment is subject to human error and is time-consuming. Machine learning-based tools have been developed to automate RoB assessment using simple models trained on limited corpuses. ChatGPT is a conversational agent based on a large language model (LLM) that was trained on an internet-scale corpus and has demonstrated human-like abilities in multiple areas including healthcare. LLMs might be able to support systematic reviewing tasks such as assessing RoB. We aim to assess interrater agreement in overall (rather than domain-level) RoB assessment between human reviewers and ChatGPT, in randomized controlled trials of interventions within medical interventions. Methods: We will randomly select 100 individually- or cluster-randomized, parallel, two-arm trials of medical interventions from recent Cochrane systematic reviews that have been assessed using the RoB1 or RoB2 family of tools. We will exclude reviews and trials that were performed under emergency conditions (e.g., COVID-19), as well as public health and welfare interventions. We will use 25 of the trials and human RoB assessments to engineer a ChatGPT prompt for assessing overall RoB, based on trial methods text. We will obtain ChatGPT assessments of RoB for the remaining 75 trials and human assessments. We will then estimate interrater agreement using Cohen’s κ. Results: The primary outcome for this study is overall human-ChatGPT interrater agreement. We will report observed agreement with an exact 95% confidence interval, expected agreement under random assessment, Cohen’s κ, and a p -value testing the null hypothesis of no difference in agreement. Several other analyses are also planned. Conclusions: This study is likely to provide the first evidence on interrater agreement between human RoB assessments and those provided by LLMs and will inform subsequent research in this area.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12874-025-02631-0

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Rose et al found moderate to substantial interrater agreement between ChatGPT-4o and human reviewers across Cochrane RoB review domains.

    “For example, Lai et al [18,27] demonstrated that LLMs can perform RoB assessments at a level comparable to trained reviewers in structured appraisal tasks, and Rose et al [25,26] found moderate to substantial interrater agreement between ChatGPT-4o and human reviewers across Cochrane RoB review domains.”

    From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [25]

    The Abstract (Results) describes agreement as an outcome the authors will report, not a finding. The Methods (Outcomes) explicitly says they will not assess agreement across Cochrane RoB domains, instead focusing on overall risk of bias. No page numbers are provided in our copy. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Recent studies have reported promising agreement between LLMs and human reviewers for RoB assessment.

    “Recent studies have examined the ability of LLMs to assist with RoB assessment and have reported promising agreement with human reviewers [18,24-27].”

    From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [25]

    The Abstract Conclusions (page unavailable) say the proposed study is likely to provide the first evidence of agreement between human and LLM risk-of-bias assessments. The Background (page unavailable) says the authors were unaware of research assessing agreement in overall risk-of-bias judgments. This protocol therefore does not report the promising agreement attributed to it. (LLM, conf=0.89)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 37 · Cited by 5

Research works printed but not linked: 26

  1. [2]Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273(5):408–12. 7823387
  2. [3]Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;18(343):d5928. PMC3196245 22008217
  3. [5]Könsgen N, Barcot O, Heß S, Puljak L, Goossen K, Rombey T, et al. Inter-review agreement of risk-of-bias judgments varied in Cochrane reviews. J Clin Epidemiol. 2020;120:25–32. 31866473
  4. [7]Blaizot A, Veettil SK, Saidoung P, Moreno-Garcia CF, Wiratunga N, Aceves-Martins M, et al. Using artificial intelligence methods for systematic review in health sciences: a systematic review. Res Synth Methods. 2022;13(3):353–62. 35174972
  5. [8]Cierco Jimenez R, Lee T, Rosillo N, Cordova R, Cree IA, Gonzalez A, et al. Machine learning computational tools to assist the performance of systematic reviews: a mapping review. BMC Med Res Methodol. 2022;22(1):322. 36522637 PMC9756658
  6. [10]Andaur Navarro CL, Damen JAA, Takada T, Nijman SWJ, Dhiman P, Ma J, et al. Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ. 2021;20(375):n2281. PMC8527348 34670780
  7. [14]Hirt J, Meichlinger J, Schumacher P, Mueller G. Agreement in risk of bias assessment between robotreviewer and human reviewers: an evaluation study on randomised controlled trials in nursing-related cochrane reviews. J Nurs Scholarsh. 2021;53(2):246–54. 33555110
  8. [15]Arno A, Thomas J, Wallace B, Marshall IJ, McKenzie JE, Elliott JH. Accuracy and efficiency of machine learning-assisted risk-of-bias assessments in “real-world” systematic reviews. Ann Intern Med. 2022;175(7):1001–9. 35635850
  9. [16]Jardim PSJ, Rose CJ, Ames HM, Echavez JFM, Van de Velde S, Muller AE. Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system. BMC Med Res Methodol. 2022;22(1):167. 35676632 PMC9174024
  10. [18]Bahor Z, Liao J, Macleod MR, Bannach-Brown A, McCann SK, Wever KE, et al. Risk of bias reporting in the recent animal focal cerebral ischaemia literature. Clin Sci. 2017;131(20):2525–32. PMC5869854 29026002
  11. [19]Wang Q, Liao J, Lapata M, Macleod M. Risk of bias assessment in preclinical literature using natural language processing. Res Synth Methods. 2022;13(3):368–80. 34709718 PMC9298308
  12. [20]Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et al. Language models are few-shot learners. Advances in neural information processing systems. 2020;33:1877–901.
  13. [25]Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit Health. 2023;2(2):e0000198. 36812645 PMC9931230
  14. [27]Bertizzolo L, Bossuyt P, Atal I, Ravaud P, Dechartres A. Disagreements in risk of bias assessment for randomised controlled trials included in more than one Cochrane systematic reviews: a research on research study using cross-sectional design. BMJ Open. 2019;9(4):e028382. PMC6500379 30940766
  15. [28]Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2013;66(9):973–81. 22981249
  16. [29]Surkov D. Using of dexmedetomidine in term neonates with hypoxic-ischemic encephalopathy. Medicni perspektivi (Medical perspectives). 2019;24(2):24–33.
  17. [30]Bäcke P, Bruschettini M, Sibrecht G, ThernströmBlomqvist Y, Olsson E. Pharmacological interventions for pain and sedation management in newborn infants undergoing therapeutic hypothermia. Cochrane Database Syst Rev. 2022;11(11):CD015023. 36354070 PMC9647594
  18. [31]Muller AE, Berg RC, Meneses-Echavez JF, Ames HMR, Borge TC, Jardim PSJ, et al. The effect of machine learning tools for evidence synthesis on resource use and time-to-completion: protocol for a retrospective pilot study. Syst Rev. 2023;12(1):7. 36650579 PMC9843684
  19. [32]Kottner J, Audigé L, Brorson S, Donner A, Gajewski BJ, Hróbjartsson A, et al. Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. J Clin Epidemiol. 2011;64(1):96–106. 21130355
  20. [34]Venkateswaran M, Ghanem B, Abbas E, Khader KA, Ward IA, Awwad T, et al. A digital health registry with clinical decision support for improving quality of antenatal care in Palestine (eRegQual): a pragmatic, cluster-randomised, controlled, superiority trial. Lancet Digit Health. 2022;4(2):e126–36. 35090675 PMC8811715
  21. [35]Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74. 843571
  22. [36]Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996;276(8):637–9. 8773637
  23. [37]Lewis JA. Statistical principles for clinical trials (ICH E9): an introductory note on an international guideline. Stat Med. 1999;18(15):1903–42. 10440877
  24. [40]Keskinbora KH. Medical ethics considerations on artificial intelligence. J Clin Neurosci. 2019;64:277–82. 30878282
  25. [42]Qiu S, Liu Q, Zhou S, Wu C. Review of artificial intelligence adversarial attack and defense technologies. Appl Sci. 2019;9(5):909.
  26. [43]Tyukin IY, Higham DJ, Gorban AN. On adversarial examples and stealth attacks in artificial intelligence systems. In: 2020 International Joint Conference on Neural Networks (IJCNN). IEEE; 2020. p. 1–6.

Authors
Rose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Berg, R. C., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses-Echavez, J. F., & Potrebny, T.
Affiliations
  • Norwegian Institute of Public Health (NO)
  • Lund University (SE)
  • University of Bristol (GB)
  • Western Norway University of Applied Sciences (NO)
Corresponding author
Rose, Christopher James
ORCID
Venue
BMC Medical Research Methodology
Publisher
Springer Science and Business Media LLC
Published
2025-07-31
Volume
25
Issue
1
Pages
182-182
ISSN
1471-2288 (ISSN-L)
DOI
10.1186/s12874-025-02631-0
PMC
PMC12315198
OpenAlex
W4412806735
Cited by
5
References
37
Citations by year
2026: 4 · 2025: 1
FWCI
3.17
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Lund University Publications (Lund University) (submittedVersion)
Funders
Norwegian Institute of Public Health
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
risk of bias assessment, ChatGPT, large language models, interobserver agreement, prompt engineering, medical interventions, randomized controlled trials, systematic review methodology
MeSH terms
Machine Learning, Systematic Reviews as Topic, Large Language Models, Generative Artificial Intelligence, Humans, Pilot Projects, Research Design, Observer Variation, Bias, Randomized Controlled Trials as Topic, Risk Assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review