Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index; 15 of the 50 sources its claims cite were checked for support.

Automated Mass Extraction of Over 680,000 PICOs from Clinical Study Abstracts Using Generative AI: A Proof-of-Concept Study.

Open access
Reason, Tim, Langham, Julia, Gimblett, Andy · 2024 · Pharmaceutical Medicine

Abstract

Generative artificial intelligence (GenAI) shows promise in automating key tasks involved in conducting systematic literature reviews (SLRs), including screening, bias assessment and data extraction. This potential automation is increasingly relevant as pharmaceutical developers face challenging requirements for timely and precise SLRs using the population, intervention, comparator and outcome (PICO) framework, such as those under the impending European Union (EU) Health Technology Assessment Regulation 2021/2282 (HTAR). This proof-of-concept study aimed to evaluate the feasibility, accuracy and efficiency of using GenAI for mass extraction of PICOs from PubMed abstracts. Abstracts were retrieved from PubMed using a search string targeting randomised controlled trials. A PubMed clinical study ‘specific/narrow’ filter was also applied. Retrieved abstracts were processed using the OpenAI Batch application programming interface (API), which allowed parallel processing and interaction with Generative Pre-trained Transformer 4 Omni (GPT-4o) via custom Python scripts. PICO elements were extracted using a zero-shot prompting strategy. Results were stored in CSV files and subsequently imported into a PostgreSQL database. The PubMed search returned 682,667 abstracts. PICOs from all abstracts were extracted in < 3 h, with an average processing time of 200 s per 1000 abstracts. A total of 395,992,770 tokens were processed, with an average of 580 tokens per abstract. The total cost was $3390. On the basis of a random sample of 350 abstracts, human verification confirmed that GPT-4o accurately and comprehensively extracted 342 (98%) of all PICOs, with only outcome elements rarely missed. Using GenAI to extract PICOs from clinical study abstracts could fundamentally transform the way SLRs are conducted. By enabling pharmaceutical developers to anticipate PICO requirements, this approach allows for proactive preparation for the EU HTAR process, or other health technology assessments (HTAs), streamlining efficiency and reducing the burden of meeting these requirements. This study explored how artificial intelligence (AI) can help automate a key part of systematic literature reviews (SLRs), which are used to summarise research in healthcare. AI was used to extract specific information referred to as population, intervention, comparator and outcome (PICOs) from nearly 700,000 abstracts of clinical studies from the PubMed database. AI extracted the PICO information from all the abstracts in under 3 h, which is significantly faster than a human could do. This demonstrates that AI could save a lot of time and effort compared with using human reviewers to do the same task. The study also found that AI performed accurately in identifying the PICO elements in the abstracts, although further human verification is still required. The use of AI in this way could help pharmaceutical developers meet upcoming requirements from the European Union, which require timely and thorough reviews of clinical evidence. AI can speed up the process, reducing the burden on pharmaceutical developers and allowing them to prepare more efficiently for these assessments. However, further testing and validation is needed before using AI in this way becomes common.

Overview

CoChat Summary

TL;DR

This proof-of-concept study used GPT-4o via the OpenAI Batch API to extract PICO elements from 682,667 PubMed randomised controlled trial abstracts in under 3 h at a total cost of $3390, with 98% accuracy in a 350-abstract human-verified sample. It suggests GenAI could streamline PICO scoping for systematic literature reviews and EU HTAR/JCA preparation, though validation is limited to PubMed abstracts and English-language records.

Key findings

  • The PubMed search returned 682,667 abstracts; PICOs from all abstracts were extracted in < 3 h, with an average processing time of 200 s per 1000 abstracts. A total of 395,992,770 tokens were processed, averaging 580 tokens per abstract, at a total cost of $3390, representing a 50% cost reduction versus processing abstracts individually.
  • On the basis of a random sample of 350 abstracts, human verification confirmed GPT-4o accurately and comprehensively extracted 342 (98%) of all PICOs; population, intervention and comparator were consistently extracted correctly, while some outcome elements were rarely missed.
  • One human reviewer required approximately 8 h to review and compare PICO data for 274 abstracts; at that rate, one reviewer would need approximately 2492 working days, or at least 10 working years, to review the total number of abstracts.
  • The authors state that, to their knowledge, this is the first instance of PICO extraction achieved with this level of speed, scale and accuracy.

Methods

  • Proof-of-concept study: abstracts were retrieved from PubMed on 11 June 2024 using a search string targeting randomised controlled trials plus the PubMed clinical study ‘specific/narrow’ filter; 682,667 abstracts were retrieved and all were included, with no further filters.
  • PICO extraction used GPT-4o through the OpenAI Batch API, custom Python scripts, and a zero-shot input/output prompting strategy; prompts were iteratively refined until no further improvements were identified. Results were stored in CSV files and imported into a PostgreSQL database.
  • Accuracy was assessed by human verification of a random subsample of 350 abstracts (0.05% of the total dataset); time and cost were tracked, with cost calculated from aggregated token counts and a 50% batch API discount.

Limitations

  • The paper states that only PubMed abstracts were used because of its freely available API, limiting generalisability; abstracts are often limited to around 300 words, so PICO elements for HTA analyses may have been missed, and further testing is needed for full texts.
  • The paper states that effectiveness in specific research areas remains unclear, validation with narrower searches is necessary, and an English-language-only filter means performance on non-English-language abstracts is unknown.
  • Evident from the text: accuracy was verified in only 350 of 682,667 abstracts (0.05%), and missed outcome elements were not verified against full publications; the authors note that further large-scale verification will be essential for regulatory and HTA settings.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s40290-024-00539-6

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Summary of the PICO extraction process using GPT-4o. PICO, population, intervention, comparator, outcome; GPT-4o, Generative Pre-trained Transformer 4 Omni. *A random sample of 350 abstracts was selected for human validation
Fig. 1 Summary of the PICO extraction process using GPT-4o. PICO, population, intervention, comparator, outcome; GPT-4o, Generative Pre-trained Transformer 4 Omni. *A random sample of 350 abstracts was selected for human validationCC BY-NC · PubMed Central
Fig. 2 Example Python code for extraction of the population of interest for one abstract
Fig. 2 Example Python code for extraction of the population of interest for one abstractCC BY-NC · PubMed Central

Verification

Claims the paper makes

7 Valid · 0 Not valid · 18 Not checked · 28 Own results and interpretations

13 more claims

Create a free account to see all 53 claims and to verify the 35 references still open.

See all claims — free account
  • It may state two separate claims as one. 21
  • Probably not a substantial claim. 25

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 23 references

Research works printed but not linked: 1

  1. PubMed. PubMed user guide: clinical study categories. 2024. https://pubmed.ncbi.nlm.nih.gov/help/#clinical-study-category-filters. Accessed 27 Aug 2024.

Cited by

See all 21

Metadata

Authors
Reason, T., Langham, J., & Gimblett, A.
Corresponding author
Reason, Tim
Venue
Pharmaceutical Medicine
Published
2024-09
Volume
38
Issue
5
Pages
365-372
ISSN
  • 0265-0673 (ISSN-L)
  • 1178-2595
  • 1179-1993
OpenAlex
W4402858850
Cited by
16
References
23
Citations by year
2026: 7 · 2025: 9
FWCI
4.47
Type
journal article
Language
en
Open access
Licence
Locations
  • Pharmaceutical Medicine (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
Keywords
generative AI, GPT-4o, automated data extraction, systematic reviews, clinical trial abstracts, PubMed, health technology assessment, zero-shot prompting, drug development, OpenAI API, token cost, randomized controlled trials
MeSH terms
Abstracting and Indexing, Proof of Concept Study, Systematic Reviews as Topic, Artificial Intelligence, Humans, Technology Assessment, Biomedical, Randomized Controlled Trials as Topic, PubMed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Automated Mass Extraction of Over 680,000 PICOs from Clinical Study Abstracts Using Generative AI: A Proof-of-Concept Study. https://app.cochat.ai/papers/10.1007/s40290-024-00539-6
The paper
Reason, T., Langham, J., & Gimblett, A. (2024). Automated Mass Extraction of Over 680,000 PICOs from Clinical Study Abstracts Using Generative AI: A Proof-of-Concept Study. Pharmaceutical Medicine. https://doi.org/10.1007/s40290-024-00539-6

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.