The DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index; 26 of the 28 sources its claims cite were checked for support.
Performance of large language models and prompt engineering strategies for data extraction in systematic reviews
Open accessAbstract
Background: Systematic reviews depend on manual data extraction and synthesis, which are time-consuming and prone to human error. Although large language models (LLMs) have the potential to automate parts of this process, their accuracy, reproducibility, and efficiency across different models and prompt strategies remain insufficiently characterized. Methods: This study evaluated the performance of three LLMs, including ChatGPT-4o, Claude 3 Sonnet, and Gemini 1.5 Pro, for data extraction from trials addressing five clinical questions (CQs) in the Japanese Clinical Practice Guidelines for the Management of Sepsis and Septic Shock 2024 (J-SSCG 2024). Using portable document format files of eligible studies, LLMs extracted predefined background characteristics and clinical outcomes. Outputs generated using an original prompt were compared with those produced using chain-of-thought and self-reflection (SR) prompt strategies. Two independent reviewers assessed accuracy against a reference standard derived from manual extraction by the guideline members. Inter-session consistency across three sessions and processing time were also evaluated. Results: For background data extraction, mean no-error proportions ranged from 81.6% (ChatGPT-4o) to 92.4% (Claude 3 Sonnet) across models. For outcome data extraction, mean no-error proportions ranged from 27.8% (Gemini 1.5 Pro) to 80.7% (Claude 3 Sonnet). Missing or incorrect values accounted for most extraction errors, whereas fabricated outputs were relatively uncommon. Prompt engineering strategies resulted in only modest changes in extraction accuracy across models. Inter-session consistency ranged from 76.3% (ChatGPT-4o) to 91.3% (Gemini 1.5 Pro) for background data extraction and from 44.8% (ChatGPT-4o) to 65.6% (Claude 3 Sonnet) for outcome data extraction. Mean processing times ranged from 29.2 to 39.7 s per article for background data extraction and from 19.3 to 46.3 s for outcome data extraction using standard prompts. When SR prompts were used, processing times increased to 59.0 to 97.7 s for background data extraction and to 52.7 to 107.1 s for outcome data extraction. Conclusions: LLMs can reliably support background data extraction in systematic reviews. However, outcome data extraction remains challenging, emphasizing the continued need for human oversight. Extraction performance varied across models and prompt engineering strategies. Clinical Trial Registration : The study was registered in the University Hospital Medical Information Network (UMIN) clinical trials registry, identifier (UMIN000054461).
Overview
CoChat Summary
TL;DR
This prospective, multi-session study evaluated ChatGPT-4o, Claude 3 Sonnet, and Gemini 1.5 Pro for extracting background and outcome data from 36 sepsis trials across five clinical questions in J-SSCG 2024. LLMs performed well for background data but poorly and inconsistently for outcome data, with prompt engineering (chain-of-thought, self-reflection) yielding only modest gains.
Key findings
- Background data extraction was reliable: mean no-error proportions were 81.6% (95% CI 76.3–87.0) for ChatGPT-4o, 92.4% (95% CI 87.6–97.2) for Claude 3 Sonnet, and 90.2% (95% CI 87.1–93.2) for Gemini 1.5 Pro.
- Outcome data extraction was much weaker and model-dependent: 64.4% (95% CI 52.3–76.6) for ChatGPT-4o, 80.7% (95% CI 65.6–95.9) for Claude 3 Sonnet, and 27.8% (95% CI 3.8–51.7) for Gemini 1.5 Pro.
- Missing or incorrect data accounted for 91.4% (95% CI 88.4–94.5) of errors; fabricated outputs were relatively uncommon (e.g., 2.7% background and 4.5% outcome for ChatGPT-4o; 0.7% and 10.0% for Claude 3 Sonnet; 14.4% and 3.9% for Gemini 1.5 Pro).
- Inter-session consistency was 76.3% (ChatGPT-4o) to 91.3% (Gemini 1.5 Pro) for background data, but only 44.8% (ChatGPT-4o) to 65.6% (Claude 3 Sonnet) for outcome data. Mean processing time with standard prompts was 29.2–39.7 s (background) and 19.3–46.3 s (outcome); with SR prompts it rose to 59.0–97.7 s and 52.7–107.1 s, respectively.
Methods
- Prospective preregistered study (UMIN000054461) extracting data from 36 eligible studies across five CQs from J-SSCG 2024; each session involved 504 background cells (14 items/study) and 208 outcome cells.
- Three LLMs (ChatGPT-4o, Claude 3 Sonnet, Gemini 1.5 Pro) accessed via web interfaces with manually uploaded PDFs (OCR-converted where needed); extraction repeated across three sessions at 2–6 week intervals, using original, chain-of-thought, and self-reflection prompts.
- Accuracy assessed at the data-element level against a human-derived reference standard by two independent reviewers (third adjudicating disagreements), with errors classified as major/minor/no error and by type (missing, incorrect, fabricated, other); sensitivity analysis compared open-access vs. paywalled articles.
Limitations
- Stated: restricted to sepsis trials from a single guideline context with a limited number of studies, limiting generalizability; model versions and generation parameters (e.g., temperature) were not controlled; findings reflect model versions available May–June 2024 and may not hold after updates; cell-level accuracy was not linked to pooled meta-analytic estimates; the human reference standard itself may contain errors.
- Evident from the text: the extraction order was not randomized, and the small number of studies per CQ produced wide confidence intervals (e.g., Gemini 1.5 Pro outcome accuracy 27.8% [95% CI 3.8–51.7]), so model comparisons are imprecise.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
8 Valid · 2 Not valid · 3 Not checked · 93 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“outcome data extraction, which often involves complex statistics and more nuanced contextual information (17, 22, 24), was substantially more challenging.”
§ Performance of large language models and prompt engineering strategies for data extraction in systematic reviews > Discussion > Comparison with prior work
What the sources say
- Supportedcites [17] Sercombe, 2025, Evaluating a customized version of ChatGPT for systematic review data extraction in health research: development and usability study, JMIR Form Res source
The Discussion (Principal Findings) reports lower agreement for extracting results (77.50%) than for study and participant characteristics (about 90–95%). It explicitly says results were challenging for the GPT to extract because studies used varied analyses and some statistical results appeared only in tables. (LLM, conf=0.92)
- Partly supportedcites [22] Bianchi, 2025, Data extractions using a large language model (elicit) and human reviewers in randomized controlled trials: a systematic comparison, Cochrane Evid Synth Methods sourceExperimental — accuracy not yet measured
The Results and Conclusion (pages unavailable) show that Elicit extracted less detail for intervention effects and had difficulty with more complex data, supporting the point that this type of extraction was challenging for Elicit. The paper does not specifically attribute that difficulty to complex statistics, nor does its comparison establish that outcome extraction is generally more challenging. (LLM, conf=0.82)
- Supportedcites [24] Motzfeldt Jensen, 2025, ChatGPT-4o can serve as the second rater for data extraction in systematic reviews, PLoS One source
The Results section (Reproducibility of ChatGPT data extraction; page unavailable) reports substantially lower agreement for outcome data (78.7%) than for general, population, or intervention data (95.4%–99.3%). The Discussion also says ChatGPT-4o faced challenges in the results section. This supports the claim in the context of ChatGPT-4o extraction. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“background characteristics were extracted with high accuracy by recent LLMs (12, 14–16)”
§ Performance of large language models and prompt engineering strategies for data extraction in systematic reviews > Discussion > Comparison with prior work
What the sources say
- Supportedcites [12] Gartlehner, 2024, Data extraction for evidence synthesis using a large language model: a proof-of-concept study, Res Synth Methods source
The Results section ('Accuracy of data extractions'; page not provided) reports high accuracy for participant background characteristics: 90% for mean age and 100% for female participants. This paper tested Claude 2, one of the LLMs covered by the citing sentence’s multiple citations. (LLM, conf=0.90)
- Partly supportedcites [14] Khraisha, 2024, Can large language models replace humans in systematic reviews? Evaluating GPT-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages, Res Synth Methods sourceExperimental — accuracy not yet measured
The Methods (p. 4) say GPT-4 extracted background characteristics such as parent and child demographics and study design. The Results (p. 6) report high raw extraction accuracy for English peer-reviewed literature (.84), but caution that this may reflect chance and describe adjusted performance as only moderate. The unqualified claim of high accuracy omits that limitation. (LLM, conf=0.86)
- Supportedcites [15] Mahmoudi, 2025, Critical assessment of large language Models’ (ChatGPT) performance in data extraction for systematic reviews: exploratory study, JMIR AI source
The Abstract Results report that ChatGPT (GPT-4) correctly extracted 28 of 30 explicitly stated study-setting elements (93.3%) after prompt refinement. The Discussion’s Principal Findings likewise describe its effectiveness in extracting basic measures; the high accuracy applies to those measures, not to complex or subjective data. (LLM, conf=0.90)
- Supportedcites [16] Lai, 2025, Language models for data extraction and risk of bias assessment in complementary medicine, NPJ Digit Med source
Table 1 in the supplied copy reports 95.07% correct LLM-only extraction of baseline characteristics. This supports the claim that recent LLMs extracted background characteristics with high accuracy. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although these proof-of-concept studies have reported encouraging results and the potential to substantially reduce manual workload and human error, most evaluations have been limited to single models, small or restricted datasets, or single extraction runs (15–20).”
§ Performance of large language models and prompt engineering strategies for data extraction in systematic reviews > Background
What the sources say
- Partly supportedcites [15] Mahmoudi, 2025, Critical assessment of large language Models’ (ChatGPT) performance in data extraction for systematic reviews: exploratory study, JMIR AI sourceExperimental — accuracy not yet measured
The Methods (Overview; Abstract > Methods) describe this evaluation as using 10 studies from one review topic, one model (GPT-4), and seven prompt iterations. The Discussion (Limitations) acknowledges the single-model and single-topic scope and says reproducibility was not formally assessed. This paper exemplifies some of the claimed limitations, but it does not establish that *most* evaluations share them; its seven iterations also do not exemplify a single extraction run. (LLM, conf=0.85)
- Partly supportedcites [16] Lai, 2025, Language models for data extraction and risk of bias assessment in complementary medicine, NPJ Digit Med sourceExperimental — accuracy not yet measured
The Abstract (page unavailable) identifies this evaluation as using 107 complementary-medicine trials, so it is an example of a study using a restricted dataset. The Methods (page unavailable) say it tested two models, but the cited paper does not establish that *most* LLM data-extraction evaluations used single models, small or restricted datasets, or single extraction runs. (LLM, conf=0.76)
- Partly supportedcites [17] Sercombe, 2025, Evaluating a customized version of ChatGPT for systematic review data extraction in health research: development and usability study, JMIR Form Res sourceExperimental — accuracy not yet measured
The Introduction describes three earlier LLM data-extraction studies, each focused on a ChatGPT version. The Discussion’s Strengths and Limitations section says this study tested one custom GPT across 20 studies and calls for future test-retest evaluation. These passages illustrate the claimed limitations, but do not establish that *most* LLM evaluations had small or restricted datasets or single extraction runs. (LLM, conf=0.82)
- Partly supportedcites [18] Polak, 2024, Extracting accurate materials data from research papers with conversational language models and prompt engineering, Nat Commun sourceExperimental — accuracy not yet measured
In Results and discussion, ‘Performance evaluation and model comparison’ (page unavailable), the paper reports a quantitative bulk-modulus test using 100 relevant sentences and 1,912 irrelevant sentences containing numbers, supporting the point that this evaluation used a property-specific dataset. But it compares ChatGPT-4, ChatGPT-3.5, and LLaMA2-chat, and does not establish that most evaluations use single models, restricted datasets, or single extraction runs. (LLM, conf=0.82)
- Supportedcites [19] Liu, 2025, AI-driven evidence synthesis: data extraction of randomized controlled trials with large language models, Int J Surg source
The Methods and Results of accuracy and efficiency assessment (pages unavailable) describe an evaluation of one model, Claude, on ten randomly selected RCTs. As one of several citations attached to the sentence, this paper exemplifies an evaluation limited to a single model and a small dataset. (LLM, conf=0.88)
- Supportedcites [20] Konet, 2024, Performance of two large language models for data extraction in evidence synthesis, Res Synth Methods source
The Limitations section describes this evaluation as using a small sample of 10 articles from one systematic review, with similarly reported outcomes for a single health condition. It therefore supports the citation as an example of a small, restricted dataset; the paper does not itself establish how common such limitations are across evaluations. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For outcome data extraction, the mean no-error proportion was 64.4% (95% CI, 52.3–76.6) for ChatGPT-4o, 80.7% (95% CI, 65.6–95.9) for Claude 3 Sonnet, and 27.8% (95% CI, 3.8–51.7) for Gemini 1.5 Pro, averaged across sessions.”
§ Performance of large language models and prompt engineering strategies for data extraction in systematic reviews > Results > Accuracy of data extraction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing or incorrect values accounted for most extraction errors, whereas fabricated outputs were relatively uncommon.”
- Also stated in: § Comparison with prior work · plain
“whereas fabricated outputs were relatively uncommon.”
§ Performance of large language models and prompt engineering strategies for data extraction in systematic reviews > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Also stated in: § Comparison with prior work · plain
69 more claims
Create a free account to see all 106 claims and every source verdict.
- It may state two separate claims as one. 14
- Probably not a substantial claim. 21
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Performance of large language models and prompt engineering strategies for data extraction in systematic reviews > Discussion > Comparison with prior work
“Claude 3 Sonnet demonstrated the most consistent performance, followed by ChatGPT-4o, whereas Gemini 1.5 Pro showed greater variability.”
§ Performance of large language models and prompt engineering strategies for data extraction in systematic reviews > Results > Consistency across sessions
“For outcome data extraction (Figure 2B), the mean consistency rate averaged across CQs was 44.8% (95% CI, 30.8–58.7) for ChatGPT-4o, 65.6% (95% CI, 40.9–90.3) for Claude 3 Sonnet, and 56.9% (95% CI, 30.2–83.7) for Gemini 1.5 Pro.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Performance of large language models and prompt engineering strategies for data extraction in systematic reviews > Results > Inter-rater agreement between two reviewers
“95.9% (95% CI, 94.3–95.7) for Gemini 1.5 Pro”
§ Performance of large language models and prompt engineering strategies for data extraction in systematic reviews > Results > Inter-rater agreement between two reviewers
“For background data, inter-rater agreement was 89.9% (95% CI, 87.5–92.2) for ChatGPT-4o, 94.6% (95% CI, 91.2–98.1) for Claude 3 Sonnet, and 95.9% (95% CI, 94.3–95.7) for Gemini 1.5 Pro.”
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“Recent advances in natural language processing have led to the development of large language models (LLMs) that can process and generate complex text ( 6–8), offering new opportunities to support and streamline systematic review processes (9–11).”
Going from evidence to recommendations
Guyatt, G. H., Oxman, A. D., Kunz, R., Falck-Ytter, Y., Vist, G. E., Liberati, A., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39493.646875.ae · Cited by 1,810 · cited 1×
“Systematic reviews are fundamental to synthesizing existing evidence and informing decision-making in healthcare (1, 2).”
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Artificial Intelligence and Machine Learning in Clinical Medicine, 2023
Haug, C. J., & Drazen, J. M. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2302038 · Cited by 1,215
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · cited 3×
“In addition to the original prompt, we tested the performance of modified prompts using the following two prompt engineering strategies: Chain-of-thought (CoT): the original prompt preceded by the instruction phrase “Let’s think step by step”, encouraging the model to reason through extraction tasks (Supplementary Appe…”
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · cited 1×
“However, conventional systematic review workflows rely heavily on manual screening and data extraction, rendering them labor-intensive, time-consuming, and susceptible to human error (3–5).”
Polak, M. P., & Morgan, D. (2024) · Nature Communications · doi:10.1038/s41467-024-45914-8 · Cited by 365 · cited 2×
“Our findings further indicated that not only model selection but also prompt engineering strategies influence performance, emphasizing prompt optimization as a primary means of improving LLM outputs (18, 19).”
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269 · cited 5×
“Emerging evidence suggests that LLMs may assist data extraction tasks as a second reviewer, improving both efficiency and consistency when extracting information from structured and semi-structured research articles (12–14).”
InfluentialKhraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268
Cited by
Salem-Hernández, J., Santiago-Gadea, P. A., Pérez-Vicente, C. A., et al. (2026) · Cureus · doi:10.7759/cureus.110790
Striković, A., Wittmann, E., Krell, J., & Osterhammer, F. (2026) · MethodsX · doi:10.1016/j.mex.2026.104089
Sánchez‐Piedra, C., Heyerdahl-Viau, I., Garcia-Carpintero, E.-E., Martínez-Núñez, J. M., & Prado-Galbarro, F. J. (2026) · Medical Sciences · doi:10.3390/medsci14050533
Kővári, A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02312-x
“This case study is best seen as an emphasis on the need for context orchestration in high-stakes settings, instead of being considered as proof of superiority over the prompting technique [15, 20].”
Influential
Metadata
- Authors
- Oami, T., Okada, Y., Maeda, K., & Nakada, T.-A.
- Affiliations
- Chiba University (JP)
- Kyoto University (JP)
- Corresponding author
- Oami, Takehiko
- Venue
- Frontiers in Digital Health
- Publisher
- Frontiers Media SA
- Published
- 2026-04-29
- Volume
- 8
- Pages
- 1799623-1799623
- ISSN
- 2673-253X (ISSN-L)
- PMC
- PMC13168097
- OpenAlex
- W7159671584
- Cited by
- 4
- References
- 28
- Citations by year
- 2026: 4
- FWCI
- 6.20
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Digital Health (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- KDDI Foundation
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, prompt engineering, automated data extraction, systematic reviews, chain-of-thought prompting, GPT-4o, Gemini 1.5 Pro, inter-rater consistency, extraction accuracy, processing time, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Performance of large language models and prompt engineering strategies for data extraction in systematic reviews. https://app.cochat.ai/papers/10.3389/fdgth.2026.1799623
- The paper
- Oami, T., Okada, Y., Maeda, K., & Nakada, T.-A. (2026). Performance of large language models and prompt engineering strategies for data extraction in systematic reviews. Frontiers in Digital Health. https://doi.org/10.3389/fdgth.2026.1799623
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.