Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index; 7 of the 18 sources its claims cite were checked for support.
Abstract
BACKGROUND: The emergence of artificial intelligence, capable of human-level performance on some tasks, presents an opportunity to revolutionise development of systematic reviews and network meta-analyses (NMAs). In this pilot study, we aim to assess use of a large-language model (LLM, Generative Pre-trained Transformer 4 [GPT-4]) to automatically extract data from publications, write an R script to conduct an NMA and interpret the results. METHODS: We considered four case studies involving binary and time-to-event outcomes in two disease areas, for which an NMA had previously been conducted manually. For each case study, a Python script was developed that communicated with the LLM via application programming interface (API) calls. The LLM was prompted to extract relevant data from publications, to create an R script to be used to run the NMA and then to produce a small report describing the analysis. RESULTS: The LLM had a > 99% success rate of accurately extracting data across 20 runs for each case study and could generate R scripts that could be run end-to-end without human input. It also produced good quality reports describing the disease area, analysis conducted, results obtained and a correct interpretation of the results. CONCLUSIONS: This study provides a promising indication of the feasibility of using current generation LLMs to automate data extraction, code generation and NMA result interpretation, which could result in significant time savings and reduce human error. This is provided that routine technical checks are performed, as recommend for human-conducted analyses. Whilst not currently 100% consistent, LLMs are likely to improve with time.
Overview
CoChat Summary
TL;DR
This pilot study assessed GPT-4 for automating data extraction, R script generation, and result interpretation for network meta-analyses across four case studies. It found >99% data extraction success and R scripts that often ran with no or minor human input, with NMA results closely matching manual analyses.
Key findings
- Across 20 runs per case study, LLM data extraction success was >99%: case study 1 had no extraction failures; case study 2 was 99.1% (35 items/run); case study 3 was 99.8% (77 items/run); case study 4 was 99.7% (35 items/run). The LLM did not report incorrect data, only intermittently failed to extract data from two trials.
- R scripts ran with no or very minor amendments in all 20 runs for case studies 1 and 2, 15 runs for case study 3, and 19 runs for case study 4. All errors caused the script to fail with an error message; no script ran and generated erroneous results.
- NMA results closely matched manual analyses: e.g., hidradenitis suppurativa adalimumab odds ratio 2.84 [2.06, 3.97] manual vs 2.84 [2.06, 3.90] GPT-4; NSCLC overall survival atezolizumab hazard ratio 0.78 [0.69, 0.88] for both. Differences were within expected variability (<1%) due to Monte Carlo error.
- Reports were clearly written, and interpretation was correct in all 20 runs for each case study, with the correct best treatment and statistical significance correctly stated.
Methods
- Four case studies used previously manually conducted NMAs: hidradenitis suppurativa clinical response (6 trials) and non-small cell lung cancer overall survival base case (5 trials), overall survival sensitivity analysis (12 trials), and progression-free survival (5 trials).
- A Python script communicated with GPT-4 via API calls to extract data from publications, generate an R script using the ‘multinma’ package and Stan, and produce a short report. Temperature was set to 0, and GPT-4 was assigned the role of statistician and medical researcher.
- Each Python script was run end to end 20 times per analysis (80 runs total). Performance was assessed by comparing extracted data with human-checked data, evaluating R script runnability and correctness against manual NMAs, and qualitatively assessing the generated reports.
Limitations
- The paper states the approach was tested on a single LLM (GPT-4) and a limited number of case studies; treatment networks were relatively simple, all trials included a common treatment, and the proportional hazards assumption was not checked for time-to-event analyses.
- The paper states LLM responses are not always consistent or reproducible, token limits required chunking publications, image-based publications could not be processed by GPT-4 at the time, data extraction from tables was out of scope, and the approach is not yet robust enough for HTA methodological rigour and lacks heterogeneity assessment.
- Evident from the text: manual feasibility and heterogeneity assessments were conducted before automation, so full NMA automation was not tested; some R script errors required domain knowledge and could not be fixed without prior network knowledge (five scripts in case study 3); and the generated report was not sufficiently detailed for an HTA appraisal.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
6 Valid · 3 Not valid · 7 Not checked · 75 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Despite these quality-control measures, the processes can still be prone to error [10].”
§ Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models > Introduction
What the sources say
- Partly supportedcites [10] Mathes T, Klaßen P, Pieper D. Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Med Res Methodol. 2017;17(1):152. https://doi.org/10.1186/s12874-017-0431-4. sourceExperimental — accuracy not yet measured
The Results report data-extraction errors in systematic reviews, and the Discussion says errors remain prevalent while comparing methods that use a second reviewer for quality assurance. This supports the point that a quality-controlled review step can still be error-prone, but the paper does not establish that HTA processes generally are prone to error. Page numbers were not available in our copy. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Whilst some studies have considered use of automation or AI within HEOR [12, 13, 16], we believe that this is the first study to consider the use of an LLM to automate data extraction and NMA end to end.”
§ Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models > Discussion
What the sources say
- Not checked yetcites [12] Rueda JD, Cristancho RA, Slejko JF. Is artificial intelligence the next big thing in health economics and outcomes research? Value Outcomes Spotlight Magazine. 2019:22–4. https://www.ispor.org/docs/default-source/publications/value-outcomes-spotlight/march-april-2019/vos-heor-articles---rueda.pdf?sfvrsn=18cb16f5_0. Accessed 05 Jan 2024.
- Supportedcites [13] van Dinter R, Tekinerdogan B, Catal C. Automation of systematic literature reviews: a systematic literature review. Inf Softw Technol. 2021;136: 106589. https://doi.org/10.1016/j.infsof.2021.106589. source
The citation supports the preceding clause about automation or AI in health-related research, not the citing authors’ own assertion that their LLM study is the first to automate data extraction and NMA end to end. The source’s abstract (p. 2) describes a review of machine-learning and NLP approaches to automating systematic reviews, and its application-domains section (p. 7) reports that 60% of the evaluated tools used medical-review data. (LLM, conf=0.87)
- Not checked yetcites [16] Poldrack RA, Lu T, Beguš G. AI-assisted coding: experiments with GPT-4. 2023. https://arxiv.org/abs/2304.13187. Accessed 05 Jan 2024.
hedgedn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach worked well for the binary outcome, where the data required for the analysis in R is number at risk and number of patients affected in each arm.”
§ Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models > Methods > Prompt Development and Key Learnings
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the analysis input, different treatments were given numbers in the R script (Fig. S7 [Online Resource]) but the LLM did not always use the same numbering for the same treatment.”
§ Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models > Methods > Prompt Development and Key Learnings
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The LLM had a > 99% success rate of accurately extracting data across 20 runs for each case study and could generate R scripts that could be run end-to-end without human input.”
§ Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models > Abstract > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
38 more claims
Create a free account to see all 91 claims and to verify the 11 references still open.
- It may state two separate claims as one. 29
- Probably not a substantial claim. 25
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs can automate critical tasks for SLRs, including screening abstracts and full texts, assessing risk of bias and extracting data for use in network meta-analyses with high accuracy, all while significantly reducing the time and costs required by using human review [2, 4–7].”
From Automated Mass Extraction of Over 680,000 PICOs from Clinical Study Abstracts Using Generative AI: A Proof-of-Concept Study. (2024) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In our experience, extracting and summarising information that is already present in text is a task that LLMs excel at [7].”
From Automated Mass Extraction of Over 680,000 PICOs from Clinical Study Abstracts Using Generative AI: A Proof-of-Concept Study. (2024) · cites this paper as [7]
Citations
Cites
PD-1 and PD-L1 Immune Checkpoint Blockade to Treat Breast Cancer
Hartkopf, A. D., Taran, F.-A., Wallwiener, M., Walter, C. B., Krämer, B., Grischke, E.-M., & Brucker, S. Y. (2016) · Breast Care · doi:10.1159/000453569 · Cited by 30,845
Adverse Renal Effects of Immune Checkpoint Inhibitors: A Narrative Review
Wanchoo, R., Karam, S., Uppal, N. N., Barta, V. S., Deray, G., Devoe, C., Launay-Vacher, V., Jhaveri, K. D., & on behalf of Cancer and Kidney International Network Workgroup on Immune Checkpoint Inhibitors (2017) · American Journal of Nephrology · doi:10.1159/000455014 · Cited by 15,256
Stan : A Probabilistic Programming Language
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017) · Journal of Statistical Software · doi:10.18637/jss.v076.i01 · Cited by 7,754 · cited 1×
“Models were estimated in a Bayesian framework using Stan [20].”
MethodHerbst, R. S., Baas, P., Kim, D.-W., Felip, E., Pérez-Gracia, J. L., Han, J.-Y., Molina, J., Kim, J.-H., Arvis, C. D., Ahn, M.-J., et al. (2016) · The Lancet · doi:10.1016/s0140-6736(15)01281-7 · Cited by 6,449 · cited 1×
“3a. Photograph of poster GPT-4 was asked to extract all data from an older publication [33] and then was provided with the updated hazard ratio (hard-coded into the Python script), before R script generation Case study 3 (NSCLC OS sensitivity) GFPC 05-06 [34] Hazard ratio provided in a figure GPT-4…”
MethodChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“Including worked examples has previously been shown to improve the performance of LLMs in multi-step reasoning tasks [21].”
MethodLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 1×
“Alternatively, using open-source LLMs, for example through the ‘Ollama’ package [26, 27], would allow tighter LLM version control and allow better reproducibility of responses.”
MethodRay, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · cited 1×
“The latest LLMs, including GPT-4 [14], have been trained using vast quantities of publicly available text data to understand, generate and manipulate human language by recognising patterns and relationships in language [15].”
MethodEvidence summaries: the evolution of a rapid review approach
Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-10 · Cited by 1,232 · cited 1×
“In addition, the HTA process is time-consuming, generally taking over a year to complete, with the SLR and NMA part of this process taking several months each [8, 11].”
Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0468-4 · Cited by 1,164
Research works printed but not linked: 1
- McKenzie J, Brennan S, Ryan R, Thomson H, Johnston R. Chapter 9: summarizing study characteristics and preparing for synthesis in Cochrane handbook for systematic reviews of interventions version 64. www.training.cochrane.org/handbook. Accessed 05 Jan 2024.
Cited by
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2024) · arXiv · doi:10.48550/arxiv.2409.04600 · Cited by 156
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf063 · Cited by 154
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Collaborative large language models for automated data extraction in living systematic reviews
Khan, M. A., Ayub, U., Naqvi, S. A. A., Khakwani, K. Z. R., Sipra, Z. B. R., Raina, A., Zhou, S., He, H., Saeidi, A., Hasan, B., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae325 · Cited by 50
Fleurence, R. L., Bian, J., Wang, X., Xu, H., Dawoud, D., Higashi, M., & Chhatwal, J. (2025) · Value in Health · doi:10.1016/j.jval.2024.10.3846 · Cited by 54
Metadata
- Authors
- Reason, T., Benbow, E., Langham, J., Gimblett, A., Klijn, S. L., & Malcolm, B.
- Affiliations
- Bristol-Myers Squibb (United States) (US)
- Bristol-Myers Squibb (United Kingdom) (GB)
- Corresponding author
- Reason, Tim
- Venue
- PharmacoEconomics - Open
- Published
- 2024-02-10
- Volume
- 8
- Issue
- 2
- Pages
- 205-220
- ISSN
- 2509-4254 (ISSN-L)
- 2509-4262
- PMC
- PMC10884375
- OpenAlex
- W4391723759
- Semantic Scholar
- 194e9a28e3b8edf5bc0c6a178a1173482ca1a2dc
- Cited by
- 71
- References
- 20
- Citations by year
- 2026: 23 · 2025: 36 · 2024: 12
- FWCI
- 19.75
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc
- creativecommons.org/licenses/by-nc/4.0 (tdm)
- creativecommons.org/licenses/by-nc/4.0 (vor)
- Locations
- PharmacoEconomics - Open (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- Bristol-Myers Squibb
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- network meta-analysis, large language models, GPT-4, automated data extraction, code generation, R code, systematic reviews, time-to-event outcomes, binary outcomes, report generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models. https://app.cochat.ai/papers/10.1007/s41669-024-00476-9
- The paper
- Reason, T., Benbow, E., Langham, J., Gimblett, A., Klijn, S. L., & Malcolm, B. (2024). Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models. PharmacoEconomics - Open. https://doi.org/10.1007/s41669-024-00476-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.