Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers
Open accessAbstract
Background: Risk of bias (RoB) assessment is a highly skilled task that is time‐consuming and subject to human error. RoB automation tools have previously used machine learning models built using relatively small task‐specific training sets. Large language models (LLMs; e.g., ChatGPT) are complex models built using non‐task‐specific Internet‐scale training sets. They demonstrate human‐like abilities and might be able to support tasks like RoB assessment. Methods: Following a published peer‐reviewed protocol, we randomly sampled 100 Cochrane reviews. New or updated reviews that evaluated medical interventions, included ≥ 1 eligible trial, and presented human consensus assessments using Cochrane RoB1 or RoB2 were eligible. We excluded reviews performed under emergency conditions (e.g., COVID‐19), and those on public health or welfare. We randomly sampled one trial from each review. Trials using individual‐ or cluster‐randomized designs were eligible. We extracted human consensus RoB assessments of the trials from the reviews, and methods texts from the trials. We used 25 review‐trial pairs to develop a ChatGPT prompt to assess RoB using trial methods text. We used the prompt and the remaining 75 review‐trial pairs to estimate human‐ChatGPT agreement for “Overall RoB” (primary outcome) and “RoB due to the randomization process”, and ChatGPT‐ChatGPT (intrarater) agreement for “Overall RoB”. We used ChatGPT‐4o (February 2025) throughout. Results: The 75 reviews were sampled from 35 Cochrane review groups, and all used RoB1. The 75 trials spanned five decades, and all but one were published in English. Human‐ChatGPT agreement for “Overall RoB” assessment was 50.7% (95% CI 39.3%–62.0%), substantially higher than expected by chance ( p = 0.0015). Human‐ChatGPT agreement for “RoB due to the randomization process” was 78.7% (95% CI 69.4%–88.0%; p < 0.001). ChatGPT‐ChatGPT agreement was 74.7% (95% CI 64.8%–84.6%; p < 0.001). Conclusions: ChatGPT appears to have some ability to assess RoB and is unlikely to be guessing or “hallucinating”. The estimated agreement for “Overall RoB” is well above estimates of agreement reported for some human reviewers, but below the highest estimates. LLM‐based systems for assessing RoB may be able to help streamline and improve evidence synthesis production.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Rose et al found moderate to substantial interrater agreement between ChatGPT-4o and human reviewers across Cochrane RoB review domains.
“For example, Lai et al [18,27] demonstrated that LLMs can perform RoB assessments at a level comparable to trained reviewers in structured appraisal tasks, and Rose et al [25,26] found moderate to substantial interrater agreement between ChatGPT-4o and human reviewers across Cochrane RoB review domains.”
From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [26]
Table 4 reports human–ChatGPT-4o agreement of 50.7% for overall RoB and 78.7% for RoB due to randomization. But its Cohen’s κ values are 0.215 (classified as fair by the paper’s criteria) and 0.537 (moderate), not moderate to substantial. The Discussion’s Strengths and Limitations section explicitly says the study did not assess other domains, so the claim overextends the finding across Cochrane RoB domains. (LLM, conf=0.91)
Experimental — accuracy not yet measuredRecent studies have reported promising agreement between LLMs and human reviewers for RoB assessment.
“Recent studies have examined the ability of LLMs to assist with RoB assessment and have reported promising agreement with human reviewers [18,24-27].”
From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817 · see citing works
RoB 2: a revised tool for assessing risk of bias in randomised trials
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., et al. (2019) · BMJ · doi:10.1136/bmj.l4898 · Cited by 33,583 · see citing works
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634 · see citing works
Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127 · see citing works
Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed
Kottner, J., Audigé, L., Brorson, S., Donner, A. P., Gajewski, B. J., Hróbjartsson, A., Roberts, C., Shoukri, M. M., & Streiner, D. L. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.002 · Cited by 2,239 · see citing works
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638 · see citing works
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472 · cited 1× · see citing works
“The role of automation in systematic review production has been explored for over a decade [8].”
Minozzi, S., Cinquini, M., Gianola, S., Gonzalez-Lorenzo, M., & Banzi, R. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.06.015 · Cited by 471 · cited 1× · see citing works
“For example, previous studies have estimated human – human agreement of 16% – 81% when conducting RoB assessments [5, 6].”
Jørgensen, L., Paludan-Müller, A. S., Laursen, D. R. T., Savović, J., Boutron, I., Sterne, J. A. C., Higgins, J. P. T., & Hróbjartsson, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0259-8 · Cited by 396 · see citing works
Living systematic reviews: 2. Combining human and machine effort
Thomas, J., Noel-Storr, A., Marshall, I., Wallace, B., McDonald, S., Mavergames, C., Glasziou, P., Shemilt, I., Synnot, A., Turner, T., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.011 · Cited by 355 · see citing works
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317 · see citing works
Page, M. J., Higgins, J. P. T., Clayton, G., Sterne, J. A. C., Hróbjartsson, A., & Savović, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0159267 · Cited by 284 · see citing works
Dechartres, A., Trinquart, L., Atal, I., Moher, D., Dickersin, K., Boutron, I., Perrodeau, E., Altman, D. G., & Ravaud, P. (2017) · BMJ · doi:10.1136/bmj.j2490 · Cited by 171 · cited 1× · see citing works
“There is evidence that humans overuse the “ uncertain ” / “ some concerns ” categories [12, 13], so we specified secondary outcomes to assess agreement between dichotomized assessments of RoB: low or unclear RoB versus high RoB, and low versus unclear or high RoB.”
Pollock, M., Fernandes, R. M., & Hartling, L. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0325-5 · Cited by 107 · see citing works
Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models
Lai, H., Ge, L., Sun, M., Pan, B., Huang, J., Hou, L., Yang, Q., Liu, J., Liu, J., Ye, Z., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.12687 · Cited by 103 · cited 1× · see citing works
“Our findings are broadly consistent with a smaller ( N = 30) study by Lai et al. [19], which published estimates of agreement all somewhat higher than ours.”
Došenović, S., Kadić, A. J., Vučić, K., Markovina, N., Pieper, D., & Puljak, L. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0493-y · Cited by 36 · see citing works
Inter-review agreement of risk-of-bias judgments varied in Cochrane reviews
Könsgen, N., Barčot, O., Heß, S., Puljak, L., Gooßen, K., Rombey, T., & Pieper, D. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.12.016 · Cited by 36 · see citing works
Šuster, S., Baldwin, T., & Verspoor, K. M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1749 · Cited by 30 · see citing works
O’Connor, A. M., Clark, J., Thomas, J., Spijker, R., Kusa, W., Walker, V. R., & Bond, M. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02666-2 · Cited by 21 · see citing works
Eisele‐Metzger, A., Lieberum, J.-L., Toews, M., Siemens, W., Heilmeyer, F., Haverkamp, C., Böhringer, D., & Meerpohl, J. J. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.12 · Cited by 21 · see citing works
Barsby, J., Hume, S., Lemmey, H. A. L., Cutteridge, J., Lee, R., & Bera, K. D. (2024) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2024-112990 · Cited by 19 · see citing works
Muller, A. E., Berg, R. C., Meneses-Echavez, J. F., Ames, H. M. R., Borge, T. C., Jardim, P. S. J., Cooper, C., & Rose, C. J. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02171-y · Cited by 17 · see citing works
Self-citationRose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Berg, R. C., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses-Echavez, J. F., & Potrebny, T. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02631-0 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Galán‐Luque, T., Quiñonero-Fernández, L., & Orgilés, M. (2026) · Children and Youth Services Review · doi:10.1016/j.childyouth.2026.108857 · Cited by 1 · see citing works
Liang, Z., Sheffield, C., Butera, G., & Antani, S. (2026) · JMIR AI · doi:10.2196/93761
Revisión sistemática con apoyo de la IA: guía metodológica y ética para investigadores
Capuñay, J. E. C., Flores, G. V. G., Márquez, G. B., Carrión, S. C. G., Alfaro, I. D. A., Quilla, I. T., Sarmiento, R. L., Talaverano, N. S. Z., Yugcha, J. D. P. H., Solar, C. Q. D., et al. (2026) · doi:10.48209/978-65-5417-668-2
Evidence‐to‐Decision Frameworks: Enhancing the Quality and Rigour of Guidelines and Recommendations
Meneses-Echavez, J. F., Alonso‐Coello, P., Vist, G., Bracchiglione, J., Song, Y., Flottorp, S. A., & Rosenbaum, S. (2026) · Clinical and Public Health Guidelines · doi:10.1002/gin2.70078
Self-citationLee, S. Y., Brown, M. D., Richards, R. J., & Smith, M. (2026) · Academic Emergency Medicine · doi:10.1111/acem.70381
Kampman, S. L., Braun, K. P. J., & Otte, W. M. (2026) · medRxiv · doi:10.64898/2026.09.09.26362643
Joseph, J. T., Vishwanath, R., & Sisirkumar, J. V. (2026) · medRxiv · doi:10.64898/2026.09.22.26363693
Adam, G. P., Davies, M. C., Caputo, E. L., Mai, H. J., Coppola, E. L., Holmer, H. K., Kuhn, E., Balk, E. M., Trikalinos, T. A., Ivlev, I., et al. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10118
Adam, G. P., Agai, E., Qureshı, R., Meman, N., Qureshi, S., Yu, W., Lam, J., & Robinson, K. A. (2026) · JMIR Bioinformatics and Biotechnology · doi:10.2196/99716
- Authors
- Rose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Potrebny, T., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses‐Echavez, J. F., & Berg, R. C.
- Affiliations
- Norwegian Institute of Public Health (NO)
- Lund University (SE)
- Glanville.info York UK
- Western Norway University of Applied Sciences (NO)
- University of Bristol (GB)
- Division of Health Services Norwegian Institute of Public Health Oslo Norway
- Corresponding author
- Rose, Christopher James
- ORCID
- Venue
- Cochrane Evidence Synthesis and Methods
- Publisher
- Wiley
- Published
- 2025-09
- Volume
- 3
- Issue
- 5
- Pages
- e70048-e70048
- ISSN
- 2832-9023 (ISSN-L)
- PMC
- PMC12442625
- OpenAlex
- W4414112257
- Cited by
- 8
- References
- 24
- Citations by year
- 2026: 8
- FWCI
- 5.12
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Cochrane Evidence Synthesis and Methods (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Bristol Research (University of Bristol) (acceptedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Norwegian Institute of Public Health
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- risk of bias assessment, GPT-4o, large language models, interobserver agreement, intraobserver agreement, randomized controlled trials, medical interventions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
