Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index; 35 of the 51 sources its claims cite were checked for support.

Automation of Systematic Reviews with Large Language Models

Open access
Cao, Christian, Arora, Rohit, Cento, Paul, Budak, Adil, Manta, Katherine, Farahani, Elina, Cecere, Matthew, Selemon, Anabel, Sang, Jason, Gong, Ling Xi, Kloosterman, Robert, Jiang, Scott, Saleh, Richard, Margalik, Denis, Lin, James, Jomy, Jane, Xie, Jerry, Chen, David, Gorla, Jaswanth, Lee, Sylvia, Zhang, Kelvin, Kuang, Jennifer, Ware, Harriet, Whelan, Mairead, Teja, Bijan, Leung, Alexander A., Arora, Rahul K., Pillay, Jennifer, Hartling, Lisa, Noetel, Michael, Emerson, David B., Detsky, Allan S., Tricco, Andrea C., Church, George M., Moher, David, Bobrovitz, Niklas · 2025 · medRxiv

Abstract

Abstract Importance Systematic reviews (SRs) inform evidence-based decision making. Yet, many take over a year to complete, are labor intensive, prone to human error, and face reproducibility challenges; thus limiting access to timely and reliable information. Objective To validate a large language model (LLM)-based workflow (otto-SR) to automate three of the most labour intensive tasks in performing SR’s: article screening, data extraction, and risk of bias assessment; and to assess its feasibility in rapidly updating existing reviews. Design, setting, and participants We conducted a validation study in four phases, with direct benchmarking against graduate-level human researchers in phases 1 and 2. Phase 1: article screening performance was measured across 32,357 citations from 5 systematic reviews. The reference standard consisted of the original reviews’ screening decisions after full-text screening. Phase 2: data extraction performance was measured across 4,495 data points from 495 studies in 7 reviews. Phase 3: risk of bias assessment (ROB2, Newcastle-Ottawa, QUADAS2) performance was measured across 345 studies from 12 reviews. Reference standards for Phase 2 and Phase 3 were created after blinded adjudication of the original review extraction and RoB assessments. Phase 4: otto-SR was used to reproduce and update the primary analysis from an issue of Cochrane reviews (n=12 reviews, 146,276 citations), with analytical comparisons to the original meta-analyzed findings. All discrepancies underwent dual human review. Results otto-SR showed high performance in phase 1 article screening ( otto-SR : 96.7% sensitivity, 97.9% specificity; human: 81.7% sensitivity, 98.1% specificity) and phase 2 data extraction ( otto-SR : 93.1% accuracy; human: 79.7% accuracy). In phase 3, otto-SR demonstrated high interrater reliability for risk of bias judgements (ROB2 0.98, Newcastle-Ottawa 0.95, QUADAS2 0.74; Gwet AC2). In phase 4, otto-SR , reproduced and updated the primary analysis from an issue of Cochrane reviews. Across Cochrane reviews, otto-SR incorrectly excluded a median of 0 studies (IQR 0 to 0.25), and found nearly twice as many eligible studies compared to the original authors (n= 114 vs. 64). Meta-analyses based on otto-SR generated screening and extraction outputs, subsequently verified through dual human review, yielded newly statistically significant effect estimates in 2 reviews and negated significance in 1 review. Conclusions and relevance LLMs have high performance in article screening, data extraction, and risk of bias assessments. They can rapidly reproduce and update existing systematic reviews, laying the foundation for automated, scalable, and reliable evidence synthesis.

Overview

CoChat Summary

## TL;DR

This paper presents otto-SR, an end-to-end LLM workflow for automating systematic review screening, data extraction, and update/reproduction tasks. Across multiple benchmarks and a 12-review Cochrane reproducibility exercise, the system outperformed human workflows on screening and extraction and reproduced/updated reviews in two days.

## Key findings

  • On screening across five reviews (32,357 citations), otto-SR achieved 96.7% sensitivity and 97.9% specificity in abstract screening, versus human dual screening at 81.7% sensitivity and 98.1% specificity; in full-text screening, otto-SR reached 97.1% sensitivity and 98.7% specificity, while dual human reviewers were at 81.3% sensitivity and 96.8% specificity in the sampled comparison.
  • On data extraction across seven reviews (4,559 data points, 495 studies), otto-SR achieved 93.1% accuracy, compared with 79.7% for dual human reviewers and 74.8% for Elicit.
  • In the Cochrane reproducibility assessment, otto-SR reproduced and updated 12 reviews from the April 2024 issue in under two days; across these reviews it incorrectly excluded a median of 0 studies (IQR 0 to 0.25) and found a median of 2.0 eligible studies (IQR 1 to 6.5) likely missed by the original authors.
  • In the updated/meta-analytic comparisons, otto-SR generated newly statistically significant conclusions in 2 reviews and negated significance in 1 review.

## Methods

  • The study built otto-SR as an end-to-end workflow using GPT-4.1 for abstract/full-text screening, Gemini 2.0 Flash for PDF-to-Markdown conversion, and o3-mini-high for data extraction.
  • Screening benchmarks used datasets from BenchSR and related sources; the authors evaluated otto-SR on the full citation sets (32,357 citations across five reviews) and compared it with dual human reviewers and Elicit against original author final inclusion decisions.
  • Data extraction benchmarks used seven reviews (including SeroTracker, PA-Outcomes, PA-Testing, Sepsis, CKD, Process, and Psyc-meds), with performance assessed using an LLM-as-a-judge framework and blinded human adjudication to create corrected ground truth.
  • For the Cochrane reproduction, the authors randomly selected the April 2024 issue, excluded 2 of 14 reviews for missing public data or irreproducible search strategy, and reproduced searches to May 8, 2025; analyses focused on each review’s primary analytical comparison and used meta-analysis in R with the metafor package.

## Limitations

  • The paper states that generalizability to other clinical questions and qualitative reviews remains to be tested.
  • It states that the LLM parser may have incorrectly extracted information from PDF articles, and that the workflow was limited to main-text content rather than supplementary tables or figures.
  • It notes that the LLM-as-a-judge framework may introduce occasional errors, and that when otto-SR and original authors agreed, the authors assumed the value was correct without further adjudication; evident from the text, this could miss shared systematic errors.

Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1101/2025.06.13.25329541

Cites

85 of 85 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

6 Valid · 9 Not valid · 5 Not checked · 207 Own results and interpretations

122 more claims · 7 with their cited sources checked

Create a free account to see all 227 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 71
  • It may state two separate claims as one. 39
  • The quote only partly states this claim. 20

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 35 references

Research works printed but not linked: 1

  1. 10.1059/0003-4819-155-8-201110180-00009 · doi:10.1059/0003-4819-155-8-201110180-00009

Cited by

See all 33

Metadata

Authors
Cao, C., Arora, R., Cento, P., Budak, A., Manta, K., Farahani, E., Cecere, M., Selemon, A., Sang, J., Gong, L. X., Kloosterman, R., Jiang, S., Saleh, R., Margalik, D., Lin, J., Jomy, J., Xie, J., Chen, D., Gorla, J., Lee, S., Zhang, K., Kuang, J., Ware, H., Whelan, M., Teja, B., Leung, A. A., Arora, R. K., Pillay, J., Hartling, L., Noetel, M., Emerson, D. B., Detsky, A. S., Tricco, A. C., Church, G. M., Moher, D., & Bobrovitz, N.
Affiliations
  • University of Toronto (CA)
  • Harvard University (US)
  • Independent researcher
  • McGill University (CA)
  • University of British Columbia (CA)
  • McGovern Institute for Brain Research, MIT, Cambridge, MA, USA
  • University of Waterloo (CA)
  • Mount Sinai Hospital (CA)
  • Mount Sinai Hospital (US)
  • St. Michael's Hospital (CA)
  • University of Calgary (CA)
  • University of Alberta (CA)
  • The University of Queensland (AU)
  • University of Ottawa (CA)
Corresponding author
Cao, Christian
Venue
medRxiv
Publisher
openRxiv
Published
2025-06-13
OpenAlex
W4411259709
Cited by
33
References
35
Citations by year
2026: 26 · 2025: 7
Type
posted content
Language
en
Open access
Locations
medRxiv (acceptedVersion, cc-by)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
systematic reviews, large language models, article screening, data extraction, risk of bias assessment, review updating, reproducibility, AI-assisted research, Newcastle-Ottawa Scale, QUADAS-2, inter-rater reliability, meta-analysis reproducibility, evidence-based decision making, human-AI collaboration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Automation of Systematic Reviews with Large Language Models. https://app.cochat.ai/papers/10.1101/2025.06.13.25329541
The paper
Cao, C., Arora, R., Cento, P., Budak, A., Manta, K., Farahani, E., Cecere, M., Selemon, A., Sang, J., Gong, L. X., Kloosterman, R., Jiang, S., Saleh, R., Margalik, D., Lin, J., Jomy, J., Xie, J., Chen, D., Gorla, J., . . . Bobrovitz, N. (2025). Automation of Systematic Reviews with Large Language Models. medRxiv. https://doi.org/10.1101/2025.06.13.25329541

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.