CoChat
Sign Up
PreprintCited by 54Meta-analysis and systematic reviewsComputational and Text Analysis MethodsTopic Modeling

Collaborative Large Language Models for Automated Data Extraction in Living Systematic Reviews

Open access
Khan, Muhammad Ali, Ayub, Umair, Naqvi, Syed Arsalan Ahmed, Khakwani, Kaneez Zahra Rubab, Sipra, Zaryab bin Riaz, Raina, Ammad, Zou, Sihan, He, Huan +13 more·2024·Journal of the American Medical Informatics Association
DOI 10.1101/2024.09.20.24314108PMC PMC11469465

Preprint — published as 10.1093/jamia/ocae325

Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: Data extraction from the published literature is the most laborious step in conducting living systematic reviews (LSRs). We aim to build a generalizable, automated data extraction workflow leveraging large language models (LLMs) that mimics the real-world two-reviewer process. Materials and Methods: A dataset of 10 clinical trials (22 publications) from a published LSR was used, focusing on 23 variables related to trial, population, and outcomes data. The dataset was split into prompt development (n=5) and held-out test sets (n=17). GPT-4-turbo and Claude-3-Opus were used for data extraction. Responses from the two LLMs were compared for concordance. In instances with discordance, original responses from each LLM were provided to the other LLM for cross-critique. Evaluation metrics, including accuracy, were used to assess performance against the manually curated gold standard. Results: In the prompt development set, 110 (96%) responses were concordant, achieving an accuracy of 0.99 against the gold standard. In the test set, 342 (87%) responses were concordant. The accuracy of the concordant responses was 0.94. The accuracy of the discordant responses was 0.41 for GPT-4-turbo and 0.50 for Claude-3-Opus. Of the 49 discordant responses, 25 (51%) became concordant after cross-critique, with an increase in accuracy to 0.76. Discussion: Concordant responses by the LLMs are likely to be accurate. In instances of discordant responses, cross-critique can further increase the accuracy. Conclusion: Large language models, when simulated in a collaborative, two-reviewer workflow, can extract data with reasonable performance, enabling truly ‘living’ systematic reviews.

AI snapshot

Written from the paper’s full text.

TL;DR

A collaborative workflow using GPT-4-turbo and Claude-3-Opus, mimicking a two-reviewer process, extracted data from clinical trial publications with high accuracy when the two models agreed. Cross-critique improved accuracy for discordant responses, supporting LLM-assisted data extraction for living systematic reviews.

Key findings

  • In the held-out test set, 342 of 391 responses (87%) were concordant between GPT-4-turbo and Claude-3-Opus; concordant responses had mean accuracy 0.94 (95% CI 0.93–0.95), precision 1.00, recall 0.92, and F1 0.96.
  • Discordant test-set responses had lower accuracy: 0.41 (95% CI 0.30–0.52) for GPT-4-turbo and 0.50 (0.36–0.64) for Claude-3-Opus. After cross-critique, 25 of 49 discordant responses (51%) became concordant, with mean accuracy 0.76 (95% CI 0.60–0.92).
  • Single-LLM test-set accuracy was 0.89 (95% CI 0.87–0.91) for GPT-4-turbo and 0.90 (0.89–0.91) for Claude-3-Opus; collaborative concordant responses had fewer hallucinations (0.25%, 95% CI 0.00–0.70%) than individual LLMs (~2.5%).
  • In the prompt development set, 110 of 115 responses (96%) were concordant, with mean accuracy 0.99 (95% CI 0.98–1.00). The average cost was $3.40 per publication for data extraction, plus $2.06 for cross-critique.

Methods

  • Used a dataset of 10 clinical trials and 22 publications from a published living systematic review and network meta-analysis on first-line treatments for metastatic castration-sensitive prostate cancer, covering 23 variables related to trial, population, and outcomes data.
  • Randomly split publications into a prompt development set (n=5) and a held-out test set (n=17); processed text with PyMuPDF and tables/figures as images, then extracted data using GPT-4-turbo and Claude-3-Opus with temperature set to zero.
  • Compared LLM responses for concordance, applied cross-critique for discordant responses by providing each LLM the other’s output, and evaluated against a manually curated gold standard using accuracy, precision, recall, F1, and hallucination percentage, with Wilcoxon signed-rank tests for comparisons.

Limitations

  • The paper states the dataset includes only prostate cancer trials, so applicability outside oncology may be limited; it also notes possible test-data contamination from including different publications from the same clinical trials in both prompt development and test sets, though sensitivity analyses showed consistent results.
  • The paper states LLMs may have been previously exposed to included publications, potentially overestimating performance, and that performance was lower for variables requiring implicit judgment; it suggests a third reviewer, human or LLM, may mitigate this.
  • Evident from the text: the evaluation is small (10 trials, 22 publications, 23 variables), uses only two proprietary LLMs, relies on manual concordance assessment, and the authors state external validation is required.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1101/2024.09.20.24314108

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 7 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    GPT-4T exhibited a mean precision of 0.29 (95% CI 0.13-0.45) on discordant responses after cross-critique in the test set.

    “From the responses that were discordant after cross-critique, GPT-4T exhibited a mean accuracy of 0.18 (95% CI: 0.06–0.30), mean precision of 0.29 (0.13–0.45), mean recall of 0.48 (0.32–0.64), and mean F1 score of 0.36 (0.20–0.52), whereas Claude-3O exhibited a mean accuracy of 0.45 (95% CI: 0.29–0.61), mean precision of 0.46 (0.30–0.62), mean recall of 0.75 (0.59–0.91) and mean F1 score of 0.57 (”
    Quote in the paperSubstantial claim
  • Findingown data

    From 5 discordant responses in the prompt development set, GPT-4T and Claude-3O generated 2 (40%) discordant responses upon cross-critique.

    “From 5 discordant responses in the prompt development set, GPT-4T and Claude-3O generated 2 (40%) discordant responses upon cross-critique.”
    Quote in the paperNot a substantial claim
  • Findingown data

    In the test set, concordant responses significantly outperformed Claude-3O in mean accuracy, precision, and F1 score.

    “and Claude-3O in terms of the mean accuracy (p-value: 0.007), precision (0.022) and F1 score (0.004).”
    Quote in the paperSubstantial claim
  • Findingown data

    In the test set, discordant responses had a mean accuracy of 0.41 for GPT-4T.

    “In discordant responses, GPT-4T exhibited a mean accuracy of 0.41 (95% CI: 0.30–0.52), mean precision of 0.54 (0.40–0.68), mean recall of 0.46 (0.32–0.60), and mean F1 score of 0.50 (0.36–0.64)”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The manually curated gold standard accurately represents the true data for the 23 variables.
  • Findingown data

    In the prompt development set, GPT-4T and Claude-3O each had a mean accuracy of 0.95.

    “For 115 responses in the prompt development set, GPT-4T exhibited a mean accuracy of 0.95 (95% CI: 0.93–0.97), mean precision of 1.00, mean recall of 0.92 (0.88–0.96), and mean F1 score of 0.96 (0.95–0.97) whereas Claude-3O exhibited a mean accuracy of 0.95 (0.93–0.97)”
    Quote in the paperNot a substantial claim
  • The quote only partly states this claim. 3
  • May combine two claims. 9
  • Classified as probably not a substantial claim. 7

  • The quote only partly states this claim.

    Findingown data

    In the test set, hallucinations were present in 26.93% of discordant responses with GPT-4T.

    “Hallucinations were present in 26.93% (95% CI: 6.38–47.48%) of the responses with GPT-4T”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the test set, concordant responses significantly outperformed GPT-4T in mean accuracy, precision, recall, and F1 score.

    “Concordant responses significantly outperform GPT-4T in terms of the mean accuracy (p-value: 0.003), precision (0.035), recall and F1 score (0.003)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The average cost for data extraction was $3.40 for one publication, with an additional cost of $2.06 for cross-critique.

    “The average cost for data extraction was $3.40 for one publication, with an additional cost of $2.06 for cross-critique in almost real-time.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    In the test set, hallucinations were present in 0.25% of concordant responses.

    “Hallucinations were present in 0.25% (95% CI: 0.00–0.70%) of the responses (Supplementary Table 20).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    In the test set, hallucinations were present in 41.00% of discordant responses with Claude-3O.

    “and 41.00% (17.41–64.59%) of the responses with Claude-3O”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The mean test set accuracy was 0.94 with collaborative LLMs, compared to 0.89 with GPT-4T and 0.90 with Claude-3O alone.

    “This is evident by a higher mean test set accuracy of 0.94 with collaborative LLMs compared to GPT-4T (0.89) and Claude-3O (0.90) alone (Figure 2).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Prompt engineering avoids the risk of overfitting associated with task-specific fine-tuning.

    “Among these strategies, prompt engineering has emerged as an effective technique to condition LLM responses[29] without the risk of overfitting associated with task-specific fine-tuning.[30]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30] Akinwande V , Jiang Y , Sam D , Kolter JZ . Understanding prompt engineering may not require rethinking generalization. arXiv preprint arXiv:2310.03957 2023.

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The dataset was randomly sampled at the publication level and split into prompt development and test sets.

    “The dataset was randomly sampled at the publication level and split into prompt development and test sets.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    This study simulated the real-world setting for data extraction in systematic reviews and showed that LLMs, when used collaboratively, analogous to the two-reviewer approach, demonstrate superior performance compared to the individual LLMs.

    “This study simulated the real-world setting for data extraction in systematic reviews and showed that LLMs, when used collaboratively, analogous to the two-reviewer approach, demonstrate superior performance compared to the individual LLMs.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The collaborative LLM workflow is analogous to the human two-reviewer approach in systematic reviews.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    A sensitivity analysis was performed on the test set excluding clinical trials that were also present in the prompt development set.

    “A sensitivity analysis was performed on the test set, excluding clinical trials that were also present in the prompt development set.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    This approach offers significant time and cost savings.

    “Third, this approach offers significant time and cost savings.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The test set was held out and not evaluated until the prompts were finalized.

    “The test set was held out and not evaluated until the prompts were finalized.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Prompt engineering is an effective technique to condition LLM responses.

    “Among these strategies, prompt engineering has emerged as an effective technique to condition LLM responses[29] without the risk of overfitting associated with task-specific fine-tuning.[30]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29] Prompt programming for large language models: Beyond the few-shot paradigm. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems; 2021.

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    This approach would significantly lower the costs and tremendously expedite the data extraction in living systematic reviews compared to manual effort.

    “Given these findings, this approach would significantly lower the costs and tremendously expedite the data extraction in LSR which otherwise would take months of manual effort[10 12].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10 12]

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The dataset used in this study includes only prostate cancer trials, and therefore the applicability to fields outside oncology may be limited.

    “The dataset used in this study includes only prostate cancer trials and hence, the applicability to the fields outside oncology may be limited.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    It is possible that the LLMs were previously exposed to publications included in the prompt development and test sets, which could have potentially overestimated the performance.

    “It is also possible that the LLMs were previously exposed to publications that were included in the prompt development and test sets, which could have potentially overestimated the performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    There is a need for more intelligent approaches for achieving automation.

    “While the collaborative approach achieved perfect accuracy for extracting variables like trial characteristics, the performance was relatively lower for other variables where implicit judgment is required, highlighting the need for more intelligent approaches for achieving automation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Large language models, when simulated in a collaborative, two-reviewer workflow, can extract data with reasonable performance, enabling truly 'living' systematic reviews.

    “Large language models, when simulated in a collaborative, two-reviewer workflow, can extract data with reasonable performance, enabling truly ‘living’ systematic reviews.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The collaborative LLM workflow is analogous to the human two-reviewer approach in systematic reviews.
    • Rests on an assumption: The laborious data extraction process is the bottleneck for creating truly ‘living’ systematic reviews.
18 of 18

  • The paper relies on this without support.

    The manually curated gold standard accurately represents the true data for the 23 variables.

    “A manually curated gold-standard dataset derived from the full-text PDFs of 22 original and follow-up publications across 10 clinical trials was used to benchmark the data extraction performance of LLMs (Supplementary Table 1).”

  • The paper relies on this without support.

    The 10 clinical trials and 22 publications from the published LSR are representative enough for the workflow to be generalizable to other LSRs.

    “A dataset of 10 clinical trials (22 publications) from a published LSR was used, focusing on 23 variables related to trial, population, and outcomes data.”

  • The paper relies on this without support.

    The collaborative LLM workflow is analogous to the human two-reviewer approach in systematic reviews.

    “This study simulated the real-world setting for data extraction in systematic reviews and showed that LLMs, when used collaboratively, analogous to the two-reviewer approach, demonstrate superior performance compared to the individual LLMs.”

  • The paper relies on this without support.

    The laborious data extraction process is the bottleneck for creating truly ‘living’ systematic reviews.

    “Given the rapid pace of evidence generation in domains like oncology[3–7] and COVID-19[41], the laborious data extraction process was the bottleneck for creating truly ‘living’ systematic reviews.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs can identify relevant literature given mechanistic context, read scientific text, and extract information into structured templates.

    “LLMs can identify relevant literature given mechanistic context, read scientific text, and extract information into structured templates [14, 15, 16, 17, 18].”

    From Structured Schemas for Provenance‐Rich, <scp>LLM</scp> ‐Assisted <scp>QSP</scp> Model Calibration (2026) · cites this paper as [15]

    The Materials and Methods sections (page not provided) describe LLMs processing text from clinical-trial publications, extracting specified variables, and formatting the results in a structured form. The paper does not show LLMs identifying relevant literature from mechanistic context; it used an existing set of publications for its extraction study. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • LLMs can identify relevant literature given mechanistic context, read scientific text, and extract information into structured templates.

    “LLMs can identify relevant literature given mechanistic context, read scientific text, and extract information into structured templates14,15,16,17,18.”

    From Structured Schemas for Provenance-Rich, LLM-Assisted QSP Model Calibration (2026) · cites this paper as [15]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Reported mean accuracy falls outside its confidence interval

    Minor

    The sensitivity-analysis mean accuracy of 0.44 is greater than the reported 95% confidence interval's upper bound of 0.18.

1 of 1

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Khan, M. A., Ayub, U., Naqvi, S. A. A., Khakwani, K. Z. R., Sipra, Z. B. R., Raina, A., Zou, S., He, H., Hossein, S. A., Hasan, B., Rumble, R. B., Bitterman, D. S., Warner, J. L., Zou, J., Tevaarwerk, A. J., Leventakos, K., Kehl, K. L., Palmer, J. M., Murad, M. H., Baral, C., & Riaz, I. B.
Affiliations
  • Mayo Clinic (US)
  • University of Arizona (US)
  • Yale University (US)
  • American Society of Clinical Oncology (US)
  • Dana-Farber Cancer Institute (US)
  • Brown University (US)
  • Arizona State University (US)
Corresponding author
Riaz, Irbaz bin
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
openRxiv
Published
2024-09-23
Volume
32
Issue
4
Pages
638-647
ISSN
1067-5027 (ISSN-L)
DOI
10.1101/2024.09.20.24314108
PMC
PMC11469465
OpenAlex
W4406663495
Semantic Scholar
4fa6de84b33bd9f0a67ecf01177935b0d9b9205f
Cited by
54
References
42
Citations by year
2026: 33 · 2025: 21
FWCI
10.23
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
Funders
  • AstraZeneca
  • National Institutes of Health
  • National Cancer Institute (U24CA265879)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
living systematic review, automated data extraction, large language models, GPT-4 Turbo, Claude 3 Opus

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review