Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Abstract
Background: The revised Risk-of-Bias tool (RoB2) overcomes the limitations of its predecessor but introduces new implementation challenges. Studies demonstrate low interrater reliability and substantial time requirements for RoB2 implementation. Large language models (LLMs) may assist in RoB2 implementation, although their effectiveness remains uncertain. Objective: This study aims to evaluate the accuracy of LLMs in RoB2 assessments to explore their potential as research assistants for bias evaluation. Methods: We systematically searched the Cochrane Library (through October 2023) for reviews using RoB2, categorized by interest in adhering or assignment. From 86 eligible reviews of randomized controlled trials (covering 1399 RCTs), we randomly selected 46 RCTs (23 per category). In addition, 3 experienced reviewers independently assessed all 46 RCTs using RoB2, recording assessment time for each trial. Reviewer judgments were reconciled through consensus. Furthermore, 6 RCTs (3 from each category) were randomly selected for prompt development and optimization. The remaining 40 trials established the internal validation standard, while Cochrane Reviews judgments served as external validation. Primary outcomes were extracted as reported in corresponding Cochrane Reviews. We calculated accuracy rates, Cohen κ, and time differentials. Results: We identified significant differences between Cochrane and reviewer judgments, particularly in domains 1, 4, and 5, likely due to different standards in assessing randomization and blinding. Among the 20 articles focusing on adhering, 18 Cochrane Reviews and 19 reviewer judgments classified them as “High risk,” while assignment-focused RCTs showed more heterogeneous risk distribution. Compared with Cochrane Reviews, LLMs demonstrated accuracy rates of 57.5% and 70% for overall (assignment) and overall (adhering), respectively. When compared with reviewer judgments, LLMs’ accuracy rates were 65% and 70% for these domains. The average accuracy rates for the remaining 6 domains were 65.2% (95% CI 57.6-72.7) against Cochrane Reviews and 74.2% (95% CI 64.7-83.9) against reviewers. At the signaling question level, LLMs achieved 83.2% average accuracy (95% CI 77.5-88.9), with accuracy exceeding 70% for most questions except 2.4 (assignment), 2.5 (assignment), 3.3, and 3.4. When domain judgments were derived from LLM-generated signaling questions using the RoB2 algorithm rather than direct LLM domain judgments, accuracy improved substantially for Domain 2 (adhering; 55-95) and overall (adhering; 70-90). LLMs demonstrated high consistency between iterations (average 85.2%, 95% CI 85.15-88.79) and completed assessments in 1.9 minutes versus 31.5 minutes for human reviewers (mean difference 29.6, 95% CI 25.6-33.6 minutes). Conclusions: LLMs achieved commendable accuracy when guided by structured prompts, particularly through processing methodological details through structured reasoning. While not replacing human assessment, LLMs demonstrate strong potential for assisting RoB2 evaluations. Larger studies with improved prompting could enhance performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“In SeroTracker, the original study authors conducted a study to assess internal consistency through re-screening a previously screened dataset using the same eligibility criteria 12 .”
From Automation of Systematic Reviews with Large Language Models (2025) · cites this paper as [12]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Dexamethasone in Hospitalized Patients with Covid-19
The RECOVERY Collaborative Group (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2021436 · Cited by 11,477
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681 · cited 1×
“During the conduct of a systematic review, assessing the risk of bias (RoB) in the included primary studies is essential, as it allows the identification of potential flaws and assessment of the internal validity of the review’s results [2].”
Remdesivir for the Treatment of Covid-19 — Final Report
Beigel, J. H., Tomashek, K. M., Dodd, L. E., Mehta, A. K., Zingman, B. S., Kalil, A. C., Hohmann, E., Chu, H. Y., Luetkemeyer, A., Kline, S., et al. (2020) · New England Journal of Medicine · doi:10.1056/nejmoa2007764 · Cited by 7,267
Bourbeau, J. (2003) · Archives of Internal Medicine · doi:10.1001/archinte.163.5.585 · Cited by 939
Novotna, A., Mares, J., Ratcliffe, S., Novakova, I., Vachova, M., Zapletalova, O., Gasperini, C., Pozzilli, C., Cefaro, L., Comi, G., et al. (2011) · European Journal of Neurology · doi:10.1111/j.1468-1331.2010.03328.x · Cited by 475
Minozzi, S., Cinquini, M., Gianola, S., Gonzalez-Lorenzo, M., & Banzi, R. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.06.015 · Cited by 471 · cited 3×
“Previous studies and reviews have shown that there are certain differences in the results of human reviewers (IRR=0.16) [4].”
InfluentialUltrafiltration Versus Usual Care for Hospitalized Patients With Heart Failure
Bart, B. A., Boyle, A., Bank, A. J., Anand, I., Olivari, M. T., Kraemer, M., Mackedanz, S., Sobotka, P. A., Schollmeyer, M., & Goldsmith, S. R. (2005) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2005.05.098 · Cited by 416
Randomized controlled trial of cannabis‐based medicine in spasticity caused by multiple sclerosis
Collin, C., Davies, P., Mutiboko, I. K., Ratcliffe, S., & for the Sativex Spasticity in MS Study Group (2007) · European Journal of Neurology · doi:10.1111/j.1468-1331.2006.01639.x · Cited by 379 · cited 1×
“We used the draft prompt to assess 6 RCTs [11-16], compared the outcomes with the criterion standard, and iteratively refined the prompt based on the responses.”
Cited by
Automation of Systematic Reviews with Large Language Models
Cao, C., Arora, R., Cento, P., Budak, A., Manta, K., Farahani, E., Cecere, M., Selemon, A., Sang, J., Gong, L. X., et al. (2025) · medRxiv · doi:10.1101/2025.06.13.25329541 · Cited by 33
Gong, E. J., Bang, C. S., & Shin, Y. S. (2026) · Bioengineering · doi:10.3390/bioengineering13030365 · Cited by 19
“Human-in-the-loop workflows that combine LLM screening with expert verification appear particularly promising, reducing assessment time by over 90% while maintaining accuracy comparable to conventional methods [55].”
JMIR Editorial Office (2025) · Journal of Medical Internet Research · doi:10.2196/80519 · Cited by 4
Oami, T., Okada, Y., & Nakada, T. (2026) · Hong Kong Journal of Emergency Medicine · doi:10.1002/hkj2.70085 · Cited by 3
Lai, H., Ma, N., Zhao, W., Liu, J., Pan, B., Tian, J., Chen, Y., Ma, B., Shang, H., Liu, J., et al. (2025) · Chinese Science Bulletin (Chinese Version) · doi:10.1360/csb-2025-5395 · Cited by 2
Self-citation
Metadata
- Authors
- Huang, J., Lai, H., Zhao, W., Xia, D., Bai, C., Sun, M., Liu, J., Liu, J., Pan, B., Tian, J., & Ge, L.
- Affiliations
- Lanzhou University (CN)
- Southern Medical University (CN)
- Peking University (CN)
- Gansu University of Traditional Chinese Medicine (CN)
- Corresponding author
- Ge, Long
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2025-06-24
- Volume
- 27
- Pages
- e70450
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC12238788
- OpenAlex
- W4410096786
- Cited by
- 20
- References
- 58
- Citations by year
- 2026: 16 · 2025: 4
- FWCI
- 12.70
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- large language models, risk of bias assessment, RoB 2, randomized controlled trials, research assistants, inter-rater reliability, blinding, assessment time, AI-assisted review
- MeSH terms
- Large Language Models, Humans, Language, Reproducibility of Results, Bias, Randomized Controlled Trials as Topic, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study. https://app.cochat.ai/papers/10.2196/70450
- The paper
- Huang, J., Lai, H., Zhao, W., Xia, D., Bai, C., Sun, M., Liu, J., Liu, J., Pan, B., Tian, J., & Ge, L. (2025). Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study. Journal of Medical Internet Research. https://doi.org/10.2196/70450
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.