CoChat

Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models

Open access

The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index; 7 of the 18 sources its claims cite were checked for support.

Abstract

BACKGROUND: The emergence of artificial intelligence, capable of human-level performance on some tasks, presents an opportunity to revolutionise development of systematic reviews and network meta-analyses (NMAs). In this pilot study, we aim to assess use of a large-language model (LLM, Generative Pre-trained Transformer 4 [GPT-4]) to automatically extract data from publications, write an R script to conduct an NMA and interpret the results. METHODS: We considered four case studies involving binary and time-to-event outcomes in two disease areas, for which an NMA had previously been conducted manually. For each case study, a Python script was developed that communicated with the LLM via application programming interface (API) calls. The LLM was prompted to extract relevant data from publications, to create an R script to be used to run the NMA and then to produce a small report describing the analysis. RESULTS: The LLM had a > 99% success rate of accurately extracting data across 20 runs for each case study and could generate R scripts that could be run end-to-end without human input. It also produced good quality reports describing the disease area, analysis conducted, results obtained and a correct interpretation of the results. CONCLUSIONS: This study provides a promising indication of the feasibility of using current generation LLMs to automate data extraction, code generation and NMA result interpretation, which could result in significant time savings and reduce human error. This is provided that routine technical checks are performed, as recommend for human-conducted analyses. Whilst not currently 100% consistent, LLMs are likely to improve with time.

Overview

CoChat Summary

TL;DR

This pilot study assessed GPT-4 for automating data extraction, R script generation, and result interpretation for network meta-analyses across four case studies. It found >99% data extraction success and R scripts that often ran with no or minor human input, with NMA results closely matching manual analyses.

Key findings

  • Across 20 runs per case study, LLM data extraction success was >99%: case study 1 had no extraction failures; case study 2 was 99.1% (35 items/run); case study 3 was 99.8% (77 items/run); case study 4 was 99.7% (35 items/run). The LLM did not report incorrect data, only intermittently failed to extract data from two trials.
  • R scripts ran with no or very minor amendments in all 20 runs for case studies 1 and 2, 15 runs for case study 3, and 19 runs for case study 4. All errors caused the script to fail with an error message; no script ran and generated erroneous results.
  • NMA results closely matched manual analyses: e.g., hidradenitis suppurativa adalimumab odds ratio 2.84 [2.06, 3.97] manual vs 2.84 [2.06, 3.90] GPT-4; NSCLC overall survival atezolizumab hazard ratio 0.78 [0.69, 0.88] for both. Differences were within expected variability (<1%) due to Monte Carlo error.
  • Reports were clearly written, and interpretation was correct in all 20 runs for each case study, with the correct best treatment and statistical significance correctly stated.

Methods

  • Four case studies used previously manually conducted NMAs: hidradenitis suppurativa clinical response (6 trials) and non-small cell lung cancer overall survival base case (5 trials), overall survival sensitivity analysis (12 trials), and progression-free survival (5 trials).
  • A Python script communicated with GPT-4 via API calls to extract data from publications, generate an R script using the ‘multinma’ package and Stan, and produce a short report. Temperature was set to 0, and GPT-4 was assigned the role of statistician and medical researcher.
  • Each Python script was run end to end 20 times per analysis (80 runs total). Performance was assessed by comparing extracted data with human-checked data, evaluating R script runnability and correctness against manual NMAs, and qualitatively assessing the generated reports.

Limitations

  • The paper states the approach was tested on a single LLM (GPT-4) and a limited number of case studies; treatment networks were relatively simple, all trials included a common treatment, and the proportional hazards assumption was not checked for time-to-event analyses.
  • The paper states LLM responses are not always consistent or reproducible, token limits required chunking publications, image-based publications could not be processed by GPT-4 at the time, data extraction from tables was out of scope, and the approach is not yet robust enough for HTA methodological rigour and lacks heterogeneity assessment.
  • Evident from the text: manual feasibility and heterogeneity assessments were conducted before automation, so full NMA automation was not tested; some R script errors required domain knowledge and could not be fixed without prior network knowledge (five scripts in case study 3); and the generated report was not sufficiently detailed for an HTA appraisal.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s41669-024-00476-9

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Overview of HTA process including NMA. Use of LLMs to automate the process have been applied within this study to those steps in the process highlighted with a red box. HTA health technology assessment, LLM large language model, NMA network meta-analysis
Fig. 1 Overview of HTA process including NMA. Use of LLMs to automate the process have been applied within this study to those steps in the process highlighted with a red box. HTA health technology assessment, LLM large language model, NMA network meta-analysisCC BY-NC · PubMed Central
Fig. 2 a LLM-based process for automating the NMA. b Chunking approach to data extraction. API application programming interface, LLM large language model, NMA network meta-analysis
Fig. 2 a LLM-based process for automating the NMA. b Chunking approach to data extraction. API application programming interface, LLM large language model, NMA network meta-analysisCC BY-NC · PubMed Central
Fig. 3 a Summary of the data extraction performance. b Summary of R script quality. HS hidradenitis suppurativa, NSCLC non-small cell lung cancer, OS overall survival, PFS progression-free survival
Fig. 3 a Summary of the data extraction performance. b Summary of R script quality. HS hidradenitis suppurativa, NSCLC non-small cell lung cancer, OS overall survival, PFS progression-free survivalCC BY-NC · PubMed Central
Fig. 4 Report writing: examples of the LLM’s (GPT-4) interpretations of the results. CrI credible interval, HR hazard ratio, HS hidradenitis suppurativa, NSCLC non-small cell cancer
Fig. 4 Report writing: examples of the LLM’s (GPT-4) interpretations of the results. CrI credible interval, HR hazard ratio, HS hidradenitis suppurativa, NSCLC non-small cell cancerCC BY-NC · PubMed Central

Verification

Claims the paper makes

6 Valid · 3 Not valid · 7 Not checked · 75 Own results and interpretations

38 more claims

Create a free account to see all 91 claims and to verify the 11 references still open.

See all claims — free account
  • It may state two separate claims as one. 29
  • Probably not a substantial claim. 25

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 23 references

Research works printed but not linked: 1

  1. McKenzie J, Brennan S, Ryan R, Thomson H, Johnston R. Chapter 9: summarizing study characteristics and preparing for synthesis in Cochrane handbook for systematic reviews of interventions version 64. www.training.cochrane.org/handbook. Accessed 05 Jan 2024.

Cited by

See all 80

Metadata

Authors
Reason, T., Benbow, E., Langham, J., Gimblett, A., Klijn, S. L., & Malcolm, B.
Affiliations
  • Bristol-Myers Squibb (United States) (US)
  • Bristol-Myers Squibb (United Kingdom) (GB)
Corresponding author
Reason, Tim
Venue
PharmacoEconomics - Open
Published
2024-02-10
Volume
8
Issue
2
Pages
205-220
ISSN
  • 2509-4254 (ISSN-L)
  • 2509-4262
OpenAlex
W4391723759
Cited by
71
References
20
Citations by year
2026: 23 · 2025: 36 · 2024: 12
FWCI
19.75
Type
journal article
Language
en
Open access
Licence
Locations
  • PharmacoEconomics - Open (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
Bristol-Myers Squibb
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
network meta-analysis, large language models, GPT-4, automated data extraction, code generation, R code, systematic reviews, time-to-event outcomes, binary outcomes, report generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models. https://app.cochat.ai/papers/10.1007/s41669-024-00476-9
The paper
Reason, T., Benbow, E., Langham, J., Gimblett, A., Klijn, S. L., & Malcolm, B. (2024). Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models. PharmacoEconomics - Open. https://doi.org/10.1007/s41669-024-00476-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.