The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index; 11 of the 24 sources its claims cite were checked for support.
Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews
Open accessAbstract
BACKGROUND: Data extraction is a critical but error-prone and labor-intensive task in evidence synthesis. Unlike other artificial intelligence (AI) technologies, large language models (LLMs) do not require labeled training data for data extraction. OBJECTIVE: To compare an AI-assisted versus a traditional, human-only data extraction process. DESIGN: Study within reviews (SWAR) using a prospective, parallel-group comparison with blinded data adjudicators. SETTING: Workflow validation within 6 ongoing systematic reviews of interventions under real-world conditions. INTERVENTION: Initial data extraction using an LLM (Claude, versions 2.1, 3.0 Opus, and 3.5 Sonnet) verified by a human reviewer. MEASUREMENTS: Concordance, time on task, accuracy, sensitivity, positive predictive value, and error analysis. RESULTS: The 6 systematic reviews in the SWAR yielded 9341 data elements from 63 studies. Concordance between the 2 methods was 77.2% (95% CI, 76.3% to 78.0%). Compared with the reference standard, the AI-assisted approach had an accuracy of 91.0% (CI, 90.4% to 91.6%) and the human-only approach an accuracy of 89.0% (CI, 88.3% to 89.6%). Sensitivities were 89.4% (CI, 88.6% to 90.1%) and 86.5% (CI, 85.7% to 87.3%), respectively, with positive predictive values of 99.2% (CI, 99.0% to 99.4%) and 98.9% (CI, 98.6% to 99.1%). Incorrect data were extracted in 9.0% (CI, 8.4% to 9.6%) of AI-assisted cases and 11.0% (CI, 10.4% to 11.7%) of human-only cases, with corresponding proportions of major errors of 2.5% (CI, 2.2% to 2.8%) versus 2.7% (CI, 2.4% to 3.1%). Missed data items were the most frequent error type in both approaches. The AI-assisted method reduced data extraction time by a median of 41 minutes per study. LIMITATIONS: Assessing concordance and classifying errors required subjective judgment. Consistently tracking time on task was challenging. CONCLUSION: Data extraction assisted by AI may offer a viable, more efficient alternative to human-only methods. PRIMARY FUNDING SOURCE: Agency for Healthcare Research and Quality and RTI International.
Overview
CoChat Summary
## TL;DR
This study-in-a-review will compare human-only versus Claude 2–assisted data extraction in systematic reviews, focusing on concordance, time, and error patterns. It is designed around extracted data elements from published study reports, with human extraction treated as the reference standard.
## Key findings
- The SWAR is powered around a prior proof-of-concept result in which Claude 2 achieved 96.3% overall accuracy on 160 data elements across 10 studies; replication accuracies were 96.9% and 95.0%.
- In that proof-of-concept, Claude 2 made 6 errors on 160 data items, and the most common error type was missed data items (n=4).
- For this SWAR, the investigators plan to assess concordance with a target precision of about ±3.5 percentage points per data category using 500 data elements per category and assuming an observed concordance of 83%.
- The authors state that, in the proof-of-concept, concordance between human extraction and LLM-assisted extraction was 83%.
## Methods
- Design: Non-randomized study within a review (SWAR) comparing two data extraction processes: human-led extraction with validation by a second investigator versus semi-automated extraction with Claude 2 followed by human validation.
- Sample/data: Publicly available published study reports in systematic reviews; the planned analysis is at the level of the data element, with categories including study characteristics, participant characteristics, interventions and participant flow, and results.
- Analysis: They will compute concordance proportions and 95% Clopper-Pearson confidence intervals, with possible exploratory subgroup analyses and, if data allow, GEE or generalized linear mixed model analyses; time will be tracked with a time-tracking app including prompt engineering time.
## Limitations
- The paper itself notes that the prior proof-of-concept was limited to open-access randomized trials, did not evaluate continuous outcome data elements, multiple study arms, or data not reported as primary outcomes.
- The reference standard is acknowledged to be imperfect: human-led data extraction is treated as the reference standard, but the text says it may need correction if errors are found.
- Evident from the text: this is a planned SWAR and the provided text does not report final results for concordance, time, or error rates.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 1 Not valid · 12 Not checked · 0 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“A randomized trial of different data extraction strategies found that single investigator data extraction and verification by a second investigator took an average of 107 minutes per study, while dual independent data extraction took 172 minutes.[2]”
What the sources say
- Supportedcites [2] 2. Li T, Saldanha IJ, Jap J, Smith BT, et al. A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews. Journal of Clinical Epidemiology 2019;115:77-89. source
The Results section (“Regular verification Independent abstraction”) reports a mean auto-recorded time of 172 minutes per article for independent abstraction. It also reports 107 minutes for regular verification, matching the other figure in the citing sentence. Section 2.7 specifies that these times cover the complete abstraction process, including adjudication where appropriate. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Previous research on methods for semi-automating data extraction has mostly focused on natural language processing (NLP) using statistical models such as naive Bayes or support vector machines.[8]”
What the sources say
- Partly supportedcites [8] 8. Schmidt L, Olorisade B, McGuinness L, et al. Data extraction methods for systematic review (semi)automation: update of a living systematic review [version 2; peer review: 3 approved]. F1000Research 2021;10:401. sourceExperimental — accuracy not yet measured
The Results section says that naïve Bayes and support vector machines were frequently used in data-extraction research. It does not establish that previous research mostly focused on those statistical models; the Discussion also describes rule-based methods as among the earliest and most frequently used approaches. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A methodology review has revealed a high rate of data extraction errors (up to 63%) in systematic reviews.[3]”
What the sources say
- Partly supportedcites [3] 3. Mathes T, Klassen P, Pieper D. Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Medical Research Methodology 2017;17:152. sourceExperimental — accuracy not yet measured
In “Studies on frequency of extraction errors,” the review reports that 63% of sampled meta-analyses had an error in at least one of two trials examined. This supports the claim that extraction errors were common, but 63% is the proportion of meta-analyses meeting that criterion, not an overall data-extraction error rate across systematic reviews. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“A randomized trial of different data extraction strategies found that single investigator data extraction and verification by a second investigator took an average of 107 minutes per study, while dual independent data extraction took 172 minutes.[2]”
What the sources say
- Supportedcites [2] 2. Li T, Saldanha IJ, Jap J, Smith BT, et al. A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews. Journal of Clinical Epidemiology 2019;115:77-89. sourceExperimental — accuracy not yet measured
Section 2.2 defines regular verification as one abstractor extracting data and a second checking it. The reported mean auto-recorded time was 107 minutes per article for regular verification, compared with 172 minutes for independent abstraction. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Therefore, research on this topic suggests that tools for automated or semi-automated data extraction based on NLP are not yet mature enough for practical use.[6]”
What the sources say
- Partly supportedcites [6] 6. Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Systematic Reviews. 2019;8:163. sourceExperimental — accuracy not yet measured
The Data extraction section (page not specified) says extraction technologies remain formative and are not readily accessible to practitioners. But it also describes semi-automatic use with a human reviewer and recommends using RobotReviewer with manual input. Thus, the source supports limited maturity, not the broader claim that these tools are not yet practical to use at all. (LLM, conf=0.87)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Partly supported
11 more claims
Create a free account to see all 21 claims and every source verdict.
- Probably not a substantial claim. 4
- It may state two separate claims as one. 2
Citations
Cites
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
The TRIPOD-LLM reporting guideline for studies using large language models
Gallifant, J., Afshar, M., Ameen, S., Aphinyanaphongs, Y., Chen, S., Cacciamani, G., Demner-Fushman, D., Dligach, D., Daneshjou, R., Fernandes, C., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03425-5 · Cited by 615
Automating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268
Mathes, T., Klaßen, P., & Pieper, D. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0431-4 · Cited by 240
Using artificial intelligence methods for systematic review in health sciences: A systematic review
Blaizot, A., Veettil, S. K., Saidoung, P., Moreno‐Garcia, C. F., Wiratunga, N., Aceves‐Martins, M., Lai, N. M., & Chaiyakunapruk, N. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1553 · Cited by 223
Quantifying Memorization Across Neural Language Models
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C. (2022) · arXiv · doi:10.48550/arxiv.2202.07646 · Cited by 204
Data extraction for evidence synthesis using a large language model: A proof-of-concept study
Gartlehner, G., Kahwati, L., Hilscher, R., Thomas, I., Kugley, S., Crotty, K., Viswanathan, M., Nussbaumer‐Streit, B., Booth, G., Erskine, N., et al. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1710 · Cited by 164
Self-citationData extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Cited by
Automation of Systematic Reviews with Large Language Models
Cao, C., Arora, R., Cento, P., Budak, A., Manta, K., Farahani, E., Cecere, M., Selemon, A., Sang, J., Gong, L. X., et al. (2025) · medRxiv · doi:10.1101/2025.06.13.25329541 · Cited by 33
Gartlehner, G., Nußbaumer-Streit, B., Hamel, C., Garritty, C., Griebler, U., King, V. J., Devane, D., & Kamel, C. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70063 · Cited by 14
Self-citationEfficient and Verified Research Data Extraction with LLM
Serdiukov, A., Dravgelis, V., Smutin, D., Taldaev, A., Ivanov, A. B., Adonin, L., & Muravyov, S. (2026) · Algorithms · doi:10.3390/a19030214 · Cited by 5
“These include missed data items (the most common error) [10], false positives/negatives in entity recognition [2,10], and poor quantitative data extraction despite good performance on categorical data [6].”
Tsuda, H., Miyazato, Y., Nakao, S., et al. (2026) · Thrombosis Journal · doi:10.1186/s12959-026-00889-y · Cited by 2
Vivekanantha, P., Kahlon, H., Balogun, O. T., Bouchard, M. D., Sa, D., Ayeni, O. R., & Kay, J. (2026) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1002/ksa.70412 · Cited by 2
“The AI ‐ assisted approach had an accuracy of 91% with a reduced extraction time of a median of 41 minutes per study [8].”
Metadata
- Authors
- Gartlehner, G., Kugley, S., Crotty, K., Viswanathan, M., Dobrescu, A., Nussbaumer-Streit, B., Booth, G., Treadwell, J. R., Han, J. M., Wagner, J., Apaydin, E. A., Coppola, E. L., Maglione, M., Hilscher, R., Chew, R., Pilar, M., Swanton, B., & Kahwati, L. C.
- Affiliations
- Universität für Weiterbildung Krems (AT)
- RTI International (US)
- ECRI Institute (US)
- University of California, Los Angeles (US)
- Kaiser Permanente (US)
- University of Southern California (US)
- ORCID
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2025-12
- Volume
- 178
- Issue
- 12
- Pages
- 1763-1771
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W4415916253
- Cited by
- 36
- References
- 17
- Citations by year
- 2026: 35 · 2025: 1
- FWCI
- 22.83
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- other-oa
- www.acpjournals.org/journal/aim/text-and-data-mining (tdm)
- Locations
- Annals of Internal Medicine (publishedVersion, other-oa)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- PubMed Central (submittedVersion)
- Funders
- U.S. Department of Veterans Affairs (IK2 HX003534)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- large language models, data extraction, systematic reviews, evidence synthesis, human-AI comparison, extraction accuracy, extraction time, AI errors, workflow validation, real-world environments, intervention, AI governance
- MeSH terms
- Systematic Reviews as Topic, Large Language Models, Artificial Intelligence, Humans, Language, Prospective Studies, Information Storage and Retrieval, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews. https://app.cochat.ai/papers/10.7326/annals-25-00739
- The paper
- Gartlehner, G., Kugley, S., Crotty, K., Viswanathan, M., Dobrescu, A., Nussbaumer-Streit, B., Booth, G., Treadwell, J. R., Han, J. M., Wagner, J., Apaydin, E. A., Coppola, E. L., Maglione, M., Hilscher, R., Chew, R., Pilar, M., Swanton, B., & Kahwati, L. C. (2025). Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews. Annals of Internal Medicine. https://doi.org/10.7326/annals-25-00739
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.