Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 26 of the 28 sources its claims cite were checked for support.

Performance of large language models and prompt engineering strategies for data extraction in systematic reviews

Open access
Oami, Takehiko, Okada, Yohei, Maeda, Kenjiro, Nakada, Taka-aki · 2026 · Frontiers in Digital Health

Abstract

Background: Systematic reviews depend on manual data extraction and synthesis, which are time-consuming and prone to human error. Although large language models (LLMs) have the potential to automate parts of this process, their accuracy, reproducibility, and efficiency across different models and prompt strategies remain insufficiently characterized. Methods: This study evaluated the performance of three LLMs, including ChatGPT-4o, Claude 3 Sonnet, and Gemini 1.5 Pro, for data extraction from trials addressing five clinical questions (CQs) in the Japanese Clinical Practice Guidelines for the Management of Sepsis and Septic Shock 2024 (J-SSCG 2024). Using portable document format files of eligible studies, LLMs extracted predefined background characteristics and clinical outcomes. Outputs generated using an original prompt were compared with those produced using chain-of-thought and self-reflection (SR) prompt strategies. Two independent reviewers assessed accuracy against a reference standard derived from manual extraction by the guideline members. Inter-session consistency across three sessions and processing time were also evaluated. Results: For background data extraction, mean no-error proportions ranged from 81.6% (ChatGPT-4o) to 92.4% (Claude 3 Sonnet) across models. For outcome data extraction, mean no-error proportions ranged from 27.8% (Gemini 1.5 Pro) to 80.7% (Claude 3 Sonnet). Missing or incorrect values accounted for most extraction errors, whereas fabricated outputs were relatively uncommon. Prompt engineering strategies resulted in only modest changes in extraction accuracy across models. Inter-session consistency ranged from 76.3% (ChatGPT-4o) to 91.3% (Gemini 1.5 Pro) for background data extraction and from 44.8% (ChatGPT-4o) to 65.6% (Claude 3 Sonnet) for outcome data extraction. Mean processing times ranged from 29.2 to 39.7 s per article for background data extraction and from 19.3 to 46.3 s for outcome data extraction using standard prompts. When SR prompts were used, processing times increased to 59.0 to 97.7 s for background data extraction and to 52.7 to 107.1 s for outcome data extraction. Conclusions: LLMs can reliably support background data extraction in systematic reviews. However, outcome data extraction remains challenging, emphasizing the continued need for human oversight. Extraction performance varied across models and prompt engineering strategies. Clinical Trial Registration : The study was registered in the University Hospital Medical Information Network (UMIN) clinical trials registry, identifier (UMIN000054461).

Overview

CoChat Summary

TL;DR

This prospective, multi-session study evaluated ChatGPT-4o, Claude 3 Sonnet, and Gemini 1.5 Pro for extracting background and outcome data from 36 sepsis trials across five clinical questions in J-SSCG 2024. LLMs performed well for background data but poorly and inconsistently for outcome data, with prompt engineering (chain-of-thought, self-reflection) yielding only modest gains.

Key findings

  • Background data extraction was reliable: mean no-error proportions were 81.6% (95% CI 76.3–87.0) for ChatGPT-4o, 92.4% (95% CI 87.6–97.2) for Claude 3 Sonnet, and 90.2% (95% CI 87.1–93.2) for Gemini 1.5 Pro.
  • Outcome data extraction was much weaker and model-dependent: 64.4% (95% CI 52.3–76.6) for ChatGPT-4o, 80.7% (95% CI 65.6–95.9) for Claude 3 Sonnet, and 27.8% (95% CI 3.8–51.7) for Gemini 1.5 Pro.
  • Missing or incorrect data accounted for 91.4% (95% CI 88.4–94.5) of errors; fabricated outputs were relatively uncommon (e.g., 2.7% background and 4.5% outcome for ChatGPT-4o; 0.7% and 10.0% for Claude 3 Sonnet; 14.4% and 3.9% for Gemini 1.5 Pro).
  • Inter-session consistency was 76.3% (ChatGPT-4o) to 91.3% (Gemini 1.5 Pro) for background data, but only 44.8% (ChatGPT-4o) to 65.6% (Claude 3 Sonnet) for outcome data. Mean processing time with standard prompts was 29.2–39.7 s (background) and 19.3–46.3 s (outcome); with SR prompts it rose to 59.0–97.7 s and 52.7–107.1 s, respectively.

Methods

  • Prospective preregistered study (UMIN000054461) extracting data from 36 eligible studies across five CQs from J-SSCG 2024; each session involved 504 background cells (14 items/study) and 208 outcome cells.
  • Three LLMs (ChatGPT-4o, Claude 3 Sonnet, Gemini 1.5 Pro) accessed via web interfaces with manually uploaded PDFs (OCR-converted where needed); extraction repeated across three sessions at 2–6 week intervals, using original, chain-of-thought, and self-reflection prompts.
  • Accuracy assessed at the data-element level against a human-derived reference standard by two independent reviewers (third adjudicating disagreements), with errors classified as major/minor/no error and by type (missing, incorrect, fabricated, other); sensitivity analysis compared open-access vs. paywalled articles.

Limitations

  • Stated: restricted to sepsis trials from a single guideline context with a limited number of studies, limiting generalizability; model versions and generation parameters (e.g., temperature) were not controlled; findings reflect model versions available May–June 2024 and may not hold after updates; cell-level accuracy was not linked to pooled meta-analytic estimates; the human reference standard itself may contain errors.
  • Evident from the text: the extraction order was not randomized, and the small number of studies per CQ produced wide confidence intervals (e.g., Gemini 1.5 Pro outcome accuracy 27.8% [95% CI 3.8–51.7]), so model comparisons are imprecise.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/fdgth.2026.1799623

Cites

30 of 31 references verified1 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Proportion of no, minor, and major errors in data extraction across large language models and prompt strategies. Each panel shows the averaged proportion of no, minor, and major errors across sessions for extraction of background (A) and outcome variables (B) by each model (ChatGPT-4o, Claude 3 Sonnet, Gemini 1.5 Pro) using the original, chain-of-thought (CoT), and self-reflection (SR) prompts.
Figure 1 Proportion of no, minor, and major errors in data extraction across large language models and prompt strategies. Each panel shows the averaged proportion of no, minor, and major errors across sessions for extraction of background (A) and outcome variables (B) by each model (ChatGPT-4o, Claude 3 Sonnet, Gemini 1.5 Pro) using the original, chain-of-thought (CoT), and self-reflection (SR) prompts.CC BY · PubMed Central
Figure 2 Inter-session consistency across large language models and prompt strategies. Each panel shows the proportion of consistent outputs for extracting background (A) and outcome variables (B) by each model (ChatGPT-4o, Claude 3 Sonnet, Gemini 1.5 Pro) using the original, chain-of-thought (CoT), and self-reflection (SR) prompts. Consistency rate (%) was calculated as the number of concordant outputs divided by the total number of outputs, multiplied by 100. Open circles represent individual consistency rates for each clinical question.
Figure 2 Inter-session consistency across large language models and prompt strategies. Each panel shows the proportion of consistent outputs for extracting background (A) and outcome variables (B) by each model (ChatGPT-4o, Claude 3 Sonnet, Gemini 1.5 Pro) using the original, chain-of-thought (CoT), and self-reflection (SR) prompts. Consistency rate (%) was calculated as the number of concordant outputs divided by the total number of outputs, multiplied by 100. Open circles represent individual consistency rates for each clinical question.CC BY · PubMed Central
Figure 3 Inter-rater agreement across large language models and prompt strategies. Inter-reviewer agreement (%) between two human reviewers, averaged across sessions, is expressed as the percentage of extraction cells for which both reviewers classified the model output identically (error/non-error). Open circles represent individual agreement rates for each clinical question. Each panel shows the agreement rate for extracting background (A) and outcome variables (B). CoT: chain-of-thought; SR: self-reflection.
Figure 3 Inter-rater agreement across large language models and prompt strategies. Inter-reviewer agreement (%) between two human reviewers, averaged across sessions, is expressed as the percentage of extraction cells for which both reviewers classified the model output identically (error/non-error). Open circles represent individual agreement rates for each clinical question. Each panel shows the agreement rate for extracting background (A) and outcome variables (B). CoT: chain-of-thought; SR: self-reflection.CC BY · PubMed Central
Figure 4 Processing time across large language models and prompt strategies. Each panel shows the processing time averaged across sessions for extracting background (A) and outcome variables (B) per article by each model (ChatGPT-4o, Claude 3 Sonnet, Gemini 1.5 Pro) using the original, chain-of-thought (CoT), and self-reflection (SR) prompts.
Figure 4 Processing time across large language models and prompt strategies. Each panel shows the processing time averaged across sessions for extracting background (A) and outcome variables (B) per article by each model (ChatGPT-4o, Claude 3 Sonnet, Gemini 1.5 Pro) using the original, chain-of-thought (CoT), and self-reflection (SR) prompts.CC BY · PubMed Central

Verification

Claims the paper makes

8 Valid · 2 Not valid · 3 Not checked · 93 Own results and interpretations

69 more claims

Create a free account to see all 106 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 14
  • Probably not a substantial claim. 21

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 28 references

Cited by

See all 4

Metadata

Authors
Oami, T., Okada, Y., Maeda, K., & Nakada, T.-A.
Affiliations
  • Chiba University (JP)
  • Kyoto University (JP)
Corresponding author
Oami, Takehiko
Venue
Frontiers in Digital Health
Published
2026-04-29
Volume
8
Pages
1799623-1799623
ISSN
2673-253X (ISSN-L)
OpenAlex
W7159671584
Cited by
4
References
28
Citations by year
2026: 4
FWCI
6.20
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Digital Health (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
KDDI Foundation
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, prompt engineering, automated data extraction, systematic reviews, chain-of-thought prompting, GPT-4o, Gemini 1.5 Pro, inter-rater consistency, extraction accuracy, processing time, reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Performance of large language models and prompt engineering strategies for data extraction in systematic reviews. https://app.cochat.ai/papers/10.3389/fdgth.2026.1799623
The paper
Oami, T., Okada, Y., Maeda, K., & Nakada, T.-A. (2026). Performance of large language models and prompt engineering strategies for data extraction in systematic reviews. Frontiers in Digital Health. https://doi.org/10.3389/fdgth.2026.1799623

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.