Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index; 11 of the 24 sources its claims cite were checked for support.

Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews

Open access

Abstract

BACKGROUND: Data extraction is a critical but error-prone and labor-intensive task in evidence synthesis. Unlike other artificial intelligence (AI) technologies, large language models (LLMs) do not require labeled training data for data extraction. OBJECTIVE: To compare an AI-assisted versus a traditional, human-only data extraction process. DESIGN: Study within reviews (SWAR) using a prospective, parallel-group comparison with blinded data adjudicators. SETTING: Workflow validation within 6 ongoing systematic reviews of interventions under real-world conditions. INTERVENTION: Initial data extraction using an LLM (Claude, versions 2.1, 3.0 Opus, and 3.5 Sonnet) verified by a human reviewer. MEASUREMENTS: Concordance, time on task, accuracy, sensitivity, positive predictive value, and error analysis. RESULTS: The 6 systematic reviews in the SWAR yielded 9341 data elements from 63 studies. Concordance between the 2 methods was 77.2% (95% CI, 76.3% to 78.0%). Compared with the reference standard, the AI-assisted approach had an accuracy of 91.0% (CI, 90.4% to 91.6%) and the human-only approach an accuracy of 89.0% (CI, 88.3% to 89.6%). Sensitivities were 89.4% (CI, 88.6% to 90.1%) and 86.5% (CI, 85.7% to 87.3%), respectively, with positive predictive values of 99.2% (CI, 99.0% to 99.4%) and 98.9% (CI, 98.6% to 99.1%). Incorrect data were extracted in 9.0% (CI, 8.4% to 9.6%) of AI-assisted cases and 11.0% (CI, 10.4% to 11.7%) of human-only cases, with corresponding proportions of major errors of 2.5% (CI, 2.2% to 2.8%) versus 2.7% (CI, 2.4% to 3.1%). Missed data items were the most frequent error type in both approaches. The AI-assisted method reduced data extraction time by a median of 41 minutes per study. LIMITATIONS: Assessing concordance and classifying errors required subjective judgment. Consistently tracking time on task was challenging. CONCLUSION: Data extraction assisted by AI may offer a viable, more efficient alternative to human-only methods. PRIMARY FUNDING SOURCE: Agency for Healthcare Research and Quality and RTI International.

Overview

CoChat Summary

## TL;DR

This study-in-a-review will compare human-only versus Claude 2–assisted data extraction in systematic reviews, focusing on concordance, time, and error patterns. It is designed around extracted data elements from published study reports, with human extraction treated as the reference standard.

## Key findings

  • The SWAR is powered around a prior proof-of-concept result in which Claude 2 achieved 96.3% overall accuracy on 160 data elements across 10 studies; replication accuracies were 96.9% and 95.0%.
  • In that proof-of-concept, Claude 2 made 6 errors on 160 data items, and the most common error type was missed data items (n=4).
  • For this SWAR, the investigators plan to assess concordance with a target precision of about ±3.5 percentage points per data category using 500 data elements per category and assuming an observed concordance of 83%.
  • The authors state that, in the proof-of-concept, concordance between human extraction and LLM-assisted extraction was 83%.

## Methods

  • Design: Non-randomized study within a review (SWAR) comparing two data extraction processes: human-led extraction with validation by a second investigator versus semi-automated extraction with Claude 2 followed by human validation.
  • Sample/data: Publicly available published study reports in systematic reviews; the planned analysis is at the level of the data element, with categories including study characteristics, participant characteristics, interventions and participant flow, and results.
  • Analysis: They will compute concordance proportions and 95% Clopper-Pearson confidence intervals, with possible exploratory subgroup analyses and, if data allow, GEE or generalized linear mixed model analyses; time will be tracked with a time-tracking app including prompt engineering time.

## Limitations

  • The paper itself notes that the prior proof-of-concept was limited to open-access randomized trials, did not evaluate continuous outcome data elements, multiple study arms, or data not reported as primary outcomes.
  • The reference standard is acknowledged to be imperfect: human-led data extraction is treated as the reference standard, but the text says it may need correction if errors are found.
  • Evident from the text: this is a planned SWAR and the provided text does not report final results for concordance, time, or error rates.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.7326/annals-25-00739

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

8 Valid · 1 Not valid · 12 Not checked · 0 Own results and interpretations

11 more claims

Create a free account to see all 21 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 4
  • It may state two separate claims as one. 2

Create a free account to see these claims.

Citations

Cites

Show all 17 references

Cited by

See all 36

Metadata

Authors
Gartlehner, G., Kugley, S., Crotty, K., Viswanathan, M., Dobrescu, A., Nussbaumer-Streit, B., Booth, G., Treadwell, J. R., Han, J. M., Wagner, J., Apaydin, E. A., Coppola, E. L., Maglione, M., Hilscher, R., Chew, R., Pilar, M., Swanton, B., & Kahwati, L. C.
Affiliations
  • Universität für Weiterbildung Krems (AT)
  • RTI International (US)
  • ECRI Institute (US)
  • University of California, Los Angeles (US)
  • Kaiser Permanente (US)
  • University of Southern California (US)
Venue
Annals of Internal Medicine
Published
2025-12
Volume
178
Issue
12
Pages
1763-1771
ISSN
  • 0003-4819 (ISSN-L)
  • 1539-3704
OpenAlex
W4415916253
Cited by
36
References
17
Citations by year
2026: 35 · 2025: 1
FWCI
22.83
Type
journal article
Language
en
Open access
Licence
  • other-oa
  • www.acpjournals.org/journal/aim/text-and-data-mining (tdm)
Locations
  • Annals of Internal Medicine (publishedVersion, other-oa)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion)
  • PubMed Central (submittedVersion)
Funders
U.S. Department of Veterans Affairs (IK2 HX003534)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
large language models, data extraction, systematic reviews, evidence synthesis, human-AI comparison, extraction accuracy, extraction time, AI errors, workflow validation, real-world environments, intervention, AI governance
MeSH terms
Systematic Reviews as Topic, Large Language Models, Artificial Intelligence, Humans, Language, Prospective Studies, Information Storage and Retrieval, Workflow

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews. https://app.cochat.ai/papers/10.7326/annals-25-00739
The paper
Gartlehner, G., Kugley, S., Crotty, K., Viswanathan, M., Dobrescu, A., Nussbaumer-Streit, B., Booth, G., Treadwell, J. R., Han, J. M., Wagner, J., Apaydin, E. A., Coppola, E. L., Maglione, M., Hilscher, R., Chew, R., Pilar, M., Swanton, B., & Kahwati, L. C. (2025). Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews. Annals of Internal Medicine. https://doi.org/10.7326/annals-25-00739

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.