Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Abstract
Background: The revised Risk-of-Bias tool (RoB2) overcomes the limitations of its predecessor but introduces new implementation challenges. Studies demonstrate low interrater reliability and substantial time requirements for RoB2 implementation. Large language models (LLMs) may assist in RoB2 implementation, although their effectiveness remains uncertain. Objective: This study aims to evaluate the accuracy of LLMs in RoB2 assessments to explore their potential as research assistants for bias evaluation. Methods: We systematically searched the Cochrane Library (through October 2023) for reviews using RoB2, categorized by interest in adhering or assignment. From 86 eligible reviews of randomized controlled trials (covering 1399 RCTs), we randomly selected 46 RCTs (23 per category). In addition, 3 experienced reviewers independently assessed all 46 RCTs using RoB2, recording assessment time for each trial. Reviewer judgments were reconciled through consensus. Furthermore, 6 RCTs (3 from each category) were randomly selected for prompt development and optimization. The remaining 40 trials established the internal validation standard, while Cochrane Reviews judgments served as external validation. Primary outcomes were extracted as reported in corresponding Cochrane Reviews. We calculated accuracy rates, Cohen κ, and time differentials. Results: We identified significant differences between Cochrane and reviewer judgments, particularly in domains 1, 4, and 5, likely due to different standards in assessing randomization and blinding. Among the 20 articles focusing on adhering, 18 Cochrane Reviews and 19 reviewer judgments classified them as “High risk,” while assignment-focused RCTs showed more heterogeneous risk distribution. Compared with Cochrane Reviews, LLMs demonstrated accuracy rates of 57.5% and 70% for overall (assignment) and overall (adhering), respectively. When compared with reviewer judgments, LLMs’ accuracy rates were 65% and 70% for these domains. The average accuracy rates for the remaining 6 domains were 65.2% (95% CI 57.6-72.7) against Cochrane Reviews and 74.2% (95% CI 64.7-83.9) against reviewers. At the signaling question level, LLMs achieved 83.2% average accuracy (95% CI 77.5-88.9), with accuracy exceeding 70% for most questions except 2.4 (assignment), 2.5 (assignment), 3.3, and 3.4. When domain judgments were derived from LLM-generated signaling questions using the RoB2 algorithm rather than direct LLM domain judgments, accuracy improved substantially for Domain 2 (adhering; 55-95) and overall (adhering; 70-90). LLMs demonstrated high consistency between iterations (average 85.2%, 95% CI 85.15-88.79) and completed assessments in 1.9 minutes versus 31.5 minutes for human reviewers (mean difference 29.6, 95% CI 25.6-33.6 minutes). Conclusions: LLMs achieved commendable accuracy when guided by structured prompts, particularly through processing methodological details through structured reasoning. While not replacing human assessment, LLMs demonstrate strong potential for assisting RoB2 evaluations. Larger studies with improved prompting could enhance performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“In SeroTracker, the original study authors conducted a study to assess internal consistency through re-screening a previously screened dataset using the same eligibility criteria 12 .”
From Automation of Systematic Reviews with Large Language Models (2025) · cites this paper as [12]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Dexamethasone in Hospitalized Patients with Covid-19
The RECOVERY Collaborative Group (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2021436 · Cited by 11,477
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681 · cited 1×
“During the conduct of a systematic review, assessing the risk of bias (RoB) in the included primary studies is essential, as it allows the identification of potential flaws and assessment of the internal validity of the review’s results [2].”
Remdesivir for the Treatment of Covid-19 — Final Report
Beigel, J. H., Tomashek, K. M., Dodd, L. E., Mehta, A. K., Zingman, B. S., Kalil, A. C., Hohmann, E., Chu, H. Y., Luetkemeyer, A., Kline, S., et al. (2020) · New England Journal of Medicine · doi:10.1056/nejmoa2007764 · Cited by 7,267
Bourbeau, J. (2003) · Archives of Internal Medicine · doi:10.1001/archinte.163.5.585 · Cited by 939
Novotna, A., Mares, J., Ratcliffe, S., Novakova, I., Vachova, M., Zapletalova, O., Gasperini, C., Pozzilli, C., Cefaro, L., Comi, G., et al. (2011) · European Journal of Neurology · doi:10.1111/j.1468-1331.2010.03328.x · Cited by 475
Minozzi, S., Cinquini, M., Gianola, S., Gonzalez-Lorenzo, M., & Banzi, R. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.06.015 · Cited by 471 · cited 3×
“Previous studies and reviews have shown that there are certain differences in the results of human reviewers (IRR=0.16) [4].”
InfluentialUltrafiltration Versus Usual Care for Hospitalized Patients With Heart Failure
Bart, B. A., Boyle, A., Bank, A. J., Anand, I., Olivari, M. T., Kraemer, M., Mackedanz, S., Sobotka, P. A., Schollmeyer, M., & Goldsmith, S. R. (2005) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2005.05.098 · Cited by 416
Randomized controlled trial of cannabis‐based medicine in spasticity caused by multiple sclerosis
Collin, C., Davies, P., Mutiboko, I. K., Ratcliffe, S., & for the Sativex Spasticity in MS Study Group (2007) · European Journal of Neurology · doi:10.1111/j.1468-1331.2006.01639.x · Cited by 379 · cited 1×
“We used the draft prompt to assess 6 RCTs [11-16], compared the outcomes with the criterion standard, and iteratively refined the prompt based on the responses.”
Fan, V. S., Gaziano, J. M., Lew, R., Bourbeau, J., Adams, S. G., Leatherman, S., Thwin, S. S., Huang, G. D., Robbins, R., Sriram, P. S., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-156-10-201205150-00003 · Cited by 338
Disease Management Program for Chronic Obstructive Pulmonary Disease: A Randomized Controlled Trial
Rice, K. L., Dewan, N., Bloomfield, H. E., Grill, J., Schult, T. M., Nelson, D. B., Kumari, S., Thomas, M., Geist, L. J., Beaner, C., et al. (2010) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.200910-1579oc · Cited by 291
Harrison, T., Oborne, J., Newton, S., & Tattersfield, A. (2004) · The Lancet · doi:10.1016/s0140-6736(03)15384-6 · Cited by 229
Doubling the dose of budesonide versus maintenance treatment in asthma exacerbations
FitzGerald, J. M. (2004) · Thorax · doi:10.1136/thx.2003.014936 · Cited by 193
Dahmen, J., Kayaalp, M. E., Ollivier, M., Pareek, A., Hirschmann, M. T., Karlsson, J., & Winkler, P. W. (2023) · Knee Surgery, Sports Traumatology, Arthroscopy · doi:10.1007/s00167-023-07355-6 · Cited by 193 · cited 1×
“Large language models (LLMs) have gradually been embraced by the medical field due to their excellent text comprehension, information extraction, and language processing capabilities, and are regarded as key technologies that could revolutionize medical practice and research [7,8].”
Anholt, R. D. V., Sobotka, L., Meijer, E. P., Heyman, H., Groen, H., Topinková, E., van Leen, M., & Schols, J. M. G. A. (2010) · Nutrition · doi:10.1016/j.nut.2010.05.009 · Cited by 159 · cited 1×
“The 20 RCTs [17-36] interested in assignment were drawn from 19 Cochrane reviews, 2 of these studies focused on COVID-19, while the other studies focused on adolescents, patients who initiated in vitro fertilization, and others.”
Minozzi, S., Dwan, K., Borrelli, F., & Filippini, G. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.09.021 · Cited by 150 · cited 3×
“Studies have shown that the time needed to assess one outcome using RoB2 can range from approximately 28 to 40 minutes [4,5].”
Quintupling Inhaled Glucocorticoids to Prevent Childhood Asthma Exacerbations
Jackson, D. J., Bacharier, L. B., Mauger, D. T., Boehmer, S., Beigelman, A., Chmiel, J. F., Fitzpatrick, A. M., Gaffin, J. M., Morgan, W. J., Peters, S. P., et al. (2018) · New England Journal of Medicine · doi:10.1056/nejmoa1710988 · Cited by 143 · cited 1×
“We used the draft prompt to assess 6 RCTs [11-16], compared the outcomes with the criterion standard, and iteratively refined the prompt based on the responses.”
Ireland, M. J., Clough, B., Gill, K., Langan, F., O'Connor, A., & Spencer, L. (2017) · Medical Teacher · doi:10.1080/0142159x.2017.1294749 · Cited by 138
Bischoff, E. W. M. A., Akkermans, R., Bourbeau, J., van Weel, C., Vercoulen, J. H., & Schermer, T. R. J. (2012) · BMJ · doi:10.1136/bmj.e7642 · Cited by 137
A self-management programme for COPD: a randomised controlled trial
Mitchell, K. E., Johnson-Warrington, V., Apps, L. D., Bankart, J., Sewell, L., Williams, J. E., Rees, K., Jolly, K., Steiner, M., Morgan, M., et al. (2014) · European Respiratory Journal · doi:10.1183/09031936.00047814 · Cited by 133
A Randomized Trial of Two Types of Nurse-Assisted Home Care for Patients With COPD
Coultas, D., Frederick, J., Barnett, B., Singh, G., & Wludyka, P. S. (2005) · CHEST Journal · doi:10.1378/chest.128.4.2017 · Cited by 122
Mental Health First Aid Training by e-Learning: A Randomized Controlled Trial
Jorm, A. F., Kitchener, B. A., Fischer, J.-A., & Cvetkovski, S. (2010) · Australian & New Zealand Journal of Psychiatry · doi:10.3109/00048674.2010.516426 · Cited by 121
Postcard intervention for repeat self-harm: randomised controlled trial.
Beautrais, A. L., Gibb, S. J., Faulkner, A., Fergusson, D. M., & Mulder, R. T. (2010) · British Journal of Psychiatry · doi:10.1192/bjp.bp.109.075754 · Cited by 120
Kawanishi, C., Aruga, T., Ishizuka, N., Yonemoto, N., Otsuka, K., Kamijo, Y., Okubo, Y., Ikeshita, K., Sakai, A., Miyaoka, H., et al. (2014) · The Lancet Psychiatry · doi:10.1016/s2215-0366(14)70259-7 · Cited by 112
Walters, J., Cameron-Tucker, H., Wills, K., Schüz, N., Scott, J., Robinson, A., Nelson, M., Turner, P., Wood-Baker, R., & Walters, E. H. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003097 · Cited by 112
Oborne, J., Mortimer, K., Hubbard, R. B., Tattersfield, A. E., & Harrison, T. W. (2009) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.200904-0616oc · Cited by 108
Clinical and Research Medical Applications of Artificial Intelligence
Ramkumar, P. N., Kunze, K. N., Haeberle, H. S., Karnuta, J. M., Luu, B. C., Nwachukwu, B. U., & Williams, R. J. (2020) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2020.08.009 · Cited by 108 · cited 1×
“Large language models (LLMs) have gradually been embraced by the medical field due to their excellent text comprehension, information extraction, and language processing capabilities, and are regarded as key technologies that could revolutionize medical practice and research [7,8].”
Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models
Lai, H., Ge, L., Sun, M., Pan, B., Huang, J., Hou, L., Yang, Q., Liu, J., Liu, J., Ye, Z., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.12687 · Cited by 103 · cited 2×
“Existing studies have confirmed that with specially designed prompting techniques, LLMs can successfully perform a modified version of the Cochrane ROB tool assessment, significantly reducing the human resources and time costs associated with traditional assessment methods [9,10].”
Self-citationPreßler, A., Christle, J. W., Lechner, B., Grabs, V., Haller, B., Hettich, I. M., Jochheim, D., Mehilli, J., Lange, R., Bleiziffer, S., et al. (2016) · American Heart Journal · doi:10.1016/j.ahj.2016.08.007 · Cited by 102 · cited 1×
“The 20 RCTs [17-36] interested in assignment were drawn from 19 Cochrane reviews, 2 of these studies focused on COVID-19, while the other studies focused on adolescents, patients who initiated in vitro fertilization, and others.”
The rationale behind systematic reviews in clinical medicine: a conceptual framework
Moosapour, H., Saeidifard, F., Aalaa, M., Soltani, A., & Larijani, B. A. (2021) · Journal of Diabetes & Metabolic Disorders · doi:10.1007/s40200-021-00773-8 · Cited by 98 · cited 1×
“They play a key role in developing guidelines and making health care decisions [1].”
Topical Fluorometholone Protects the Ocular Surface of Dry Eye Patients from Desiccating Stress
Pinto-Fraga, J., López‐Miguel, A., González-García, M. J., Fernández, I., la Rosa, A. L., Enríquez‐de‐Salamanca, A., Stern, M. E., & Calonge, M. (2015) · Ophthalmology · doi:10.1016/j.ophtha.2015.09.029 · Cited by 96
COMET: a multicomponent home-based disease-management programmeversusroutine care in severe COPD
Kessler, R., Casán, P., Koehler, D., Tognella, S., Viejo, J. L., Negro, R. W. D., Lobato, S. D., Reissig, K., González-Moro, J. M. R., Devouassoux, G., et al. (2018) · European Respiratory Journal · doi:10.1183/13993003.01612-2017 · Cited by 94
Frattarelli, J. L., Leondires, M. P., McKeeby, J. L., Miller, B. T., & Segars, J. H. (2003) · Fertility and Sterility · doi:10.1016/s0015-0282(02)04558-2 · Cited by 89
Dissanayake, E., Tani, Y., Nagai, K., Sahara, M., Mitsuishi, C., Togawa, Y., Suzuki, Y., Nakano, T., Yamaide, F., Ohno, H., et al. (2019) · International Archives of Allergy and Immunology · doi:10.1159/000501636 · Cited by 86
Botella‐Carretero, J. I., Iglesias, B., Balsa, J. A., Zamarrón, I., Arrieta, F., & Vázquez, C. (2008) · Journal of Parenteral and Enteral Nutrition · doi:10.1177/0148607108314760 · Cited by 85
Griffiths, H., Duffy, F., Duffy, L., Brown, S., Hockaday, H., Eliasson, E., Graham, J., Smith, J., Thomson, A., & Schwannauer, M. (2019) · BMC Psychiatry · doi:10.1186/s12888-019-2158-8 · Cited by 82
Singh, S., Rees, K., Gelder, C. M., Morgan, M. D., & Johnson, V. (2016) · International Journal of COPD · doi:10.2147/copd.s91253 · Cited by 78
Hernández, C., Alonso, A., García‐Aymerich, J., Serra, I., Marti, D., Rodriguez‐Roisin, R., Narsavage, G. L., Gomez, M. C., & Roca, J. (2015) · npj Primary Care Respiratory Medicine · doi:10.1038/npjpcrm.2015.22 · Cited by 77
Jónsdóttir, H., Ámundadóttir, Ó. R., Guðmundsson, G., Halldorsdottir, B. S., Hrafnkelsson, B., Ingadóttir, T. S., Jónsdóttir, R., Jónsson, J. S., Sigurjonsdottir, E. D., & Stefansdottir, I. (2015) · Journal of Advanced Nursing · doi:10.1111/jan.12728 · Cited by 71
Wang, L. H., Zhao, Y., Chen, L. Y., Zhang, L., & Zhang, Y. M. (2019) · The Clinical Respiratory Journal · doi:10.1111/crj.13112 · Cited by 61
Liljeroos, M., Ågren, S., Jaarsma, T., Årestedt, K., & Strömberg, A. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138058 · Cited by 60
Language models for data extraction and risk of bias assessment in complementary medicine
Lai, H., Liu, J., Bai, C., Liu, H., Pan, B., Luo, X., Hou, L., Zhao, W., Xia, D., Tian, J., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01457-w · Cited by 59 · cited 1×
“Existing studies have confirmed that with specially designed prompting techniques, LLMs can successfully perform a modified version of the Cochrane ROB tool assessment, significantly reducing the human resources and time costs associated with traditional assessment methods [9,10].”
Self-citationExacerbation action plans for patients with COPD and comorbidities: a randomised controlled trial
Lenferink, A., van der Palen, J. A. M., van der Valk, P. D. L. P. M., Cafarella, P., van Veen, A., Quinn, S. J., Groothuis‐Oudshoorn, C. G. M., Burt, M. G., Young, M., Frith, P. A., et al. (2019) · European Respiratory Journal · doi:10.1183/13993003.02134-2018 · Cited by 59
Efficacy of a self-management plan in exacerbations for patients with advanced COPD
Bernabéu-Mora, R., Sánchez-Nieto, J. M., Espinosa, R. A., Hu, C., Gálvez-Martínez, B., Carrillo‐Alcaraz, A., Álvarez-Miranda, C. F., Birlanga, O. M., & Abad‐Corpa, E. (2016) · International Journal of COPD · doi:10.2147/copd.s104728 · Cited by 48
Interdisciplinary COPD intervention in primary care: a cluster randomised controlled trial
Liang, J., Abramson, M. J., Russell, G. M., Holland, A. E., Zwar, N. A., Bonevski, B., Mahal, A. S., Eustace, P., Paul, E. M., Phillips, K., et al. (2019) · European Respiratory Journal · doi:10.1183/13993003.01530-2018 · Cited by 48
Rice-McDonald, G. G., Bowler, S., Staines, G., & Mitchell, C. (2005) · Internal Medicine Journal · doi:10.1111/j.1445-5994.2005.00972.x · Cited by 45
Minozzi, S., Gonzalez-Lorenzo, M., Cinquini, M., Berardinelli, D., Cagnazzo, C., Ciardullo, S., De Nardi, P., Gammone, M., Iovino, P., Lando, A., et al. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.09.003 · Cited by 41
Huhn, M., Leucht, C., Rothe, P., Dold, M., Heres, S., Bornschein, S., Schneider-Axmann, T., Hasan, A., & Leucht, S. (2021) · European Archives of Psychiatry and Clinical Neuroscience · doi:10.1007/s00406-020-01109-y · Cited by 38
Titova, E. V., Steinshamn, S., Indredavik, B., & Henriksen, A. H. (2015) · Respiratory Research · doi:10.1186/s12931-015-0170-1 · Cited by 37
Hu, F., Huang, Y., Lin, H., Chen, C., Chen, M., & Chang, C. (2020) · Geriatrics and gerontology international/Geriatrics & gerontology international · doi:10.1111/ggi.13891 · Cited by 24
Ono, T., Gotô, H., Sakai, T., Nitta, F., Mizuki, N., Takase, H., Kaneko, Y., Hori, J., Nakano, S., Nao‐i, N., et al. (2021) · Japanese Journal of Ophthalmology · doi:10.1007/s10384-021-00878-w · Cited by 23
Ooi, D. S. Q., Ling, J. Q. R., Sadananthan, S. A., Velan, S. S., Ong, F. Y., Khoo, C. M., Tai, E. S., Henry, C. J., Leow, M. K., Khoo, E. Y. H., et al. (2020) · Journal of Nutrition · doi:10.1093/jn/nxaa414 · Cited by 22
González–Ortega, I., Vega, P., Echeburúa, E., Alberich, S., Fernández-Sevillano, J., Barbeito, S., Balanzá‐Martínez, V., Vieta, E., Lorente-Rovira, E., Luengo, A., et al. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18147239 · Cited by 20
Pregnancy outcome after blastocyst stage transfer comparing to early cleavage stage embryo transfer
Aziminekoo, E., Salehi, M. S. M., Kalantari, V., Tehraninejad, E. S., Haghollahi, F., Rashidi, B. H., & Zandieh, Z. (2015) · Gynecological Endocrinology · doi:10.3109/09513590.2015.1056141 · Cited by 18
Tauber, S., Cupp, G., Garber, R. L., Bartell, J. G., Vohra, F. A., & Stroman, D. W. (2011) · Advances in Therapy · doi:10.1007/s12325-011-0037-x · Cited by 18
Curran, M., Tierney, A. C., Collins, L., Kennedy, L., McDonnell, C., Jurascheck, A. J., Sheikhi, A., Walsh, C., Button, B., Casserly, B., et al. (2023) · Journal of Cystic Fibrosis · doi:10.1016/j.jcf.2022.11.002 · Cited by 15
(2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-156-10-201205150-00001 · Cited by 13
Cited by
Automation of Systematic Reviews with Large Language Models
Cao, C., Arora, R., Cento, P., Budak, A., Manta, K., Farahani, E., Cecere, M., Selemon, A., Sang, J., Gong, L. X., et al. (2025) · medRxiv · doi:10.1101/2025.06.13.25329541 · Cited by 33
Gong, E. J., Bang, C. S., & Shin, Y. S. (2026) · Bioengineering · doi:10.3390/bioengineering13030365 · Cited by 19
“Human-in-the-loop workflows that combine LLM screening with expert verification appear particularly promising, reducing assessment time by over 90% while maintaining accuracy comparable to conventional methods [55].”
JMIR Editorial Office (2025) · Journal of Medical Internet Research · doi:10.2196/80519 · Cited by 4
Oami, T., Okada, Y., & Nakada, T. (2026) · Hong Kong Journal of Emergency Medicine · doi:10.1002/hkj2.70085 · Cited by 3
Lai, H., Ma, N., Zhao, W., Liu, J., Pan, B., Tian, J., Chen, Y., Ma, B., Shang, H., Liu, J., et al. (2025) · Chinese Science Bulletin (Chinese Version) · doi:10.1360/csb-2025-5395 · Cited by 2
Self-citation
Metadata
- Authors
- Huang, J., Lai, H., Zhao, W., Xia, D., Bai, C., Sun, M., Liu, J., Liu, J., Pan, B., Tian, J., & Ge, L.
- Affiliations
- Lanzhou University (CN)
- Southern Medical University (CN)
- Peking University (CN)
- Gansu University of Traditional Chinese Medicine (CN)
- Corresponding author
- Ge, Long
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2025-06-24
- Volume
- 27
- Pages
- e70450
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC12238788
- OpenAlex
- W4410096786
- Cited by
- 20
- References
- 58
- Citations by year
- 2026: 16 · 2025: 4
- FWCI
- 12.70
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- large language models, risk of bias assessment, RoB 2, randomized controlled trials, research assistants, inter-rater reliability, blinding, assessment time, AI-assisted review
- MeSH terms
- Large Language Models, Humans, Language, Reproducibility of Results, Bias, Randomized Controlled Trials as Topic, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.