The DOI matches its Crossref record; the references it cites are not checked yet.
Evaluating a Customized Version of ChatGPT for Systematic Review Data Extraction in Health Research: Development and Usability Study
Open accessAbstract
Background: Systematic reviews are essential for synthesizing research in health sciences; however, they are resource-intensive and prone to human error. The data extraction phase, in which key details of studies are identified and recorded in a systematic manner, may benefit from the application of automation processes. Recent advancements in artificial intelligence, specifically in large language models (LLMs) such as ChatGPT, may streamline this process. Objective: This study aimed to develop and evaluate a custom Generative Pre-Training Transformer (GPT), named Systematic Review Extractor Pro , for automating the data extraction phase of systematic reviews in health research. Methods: OpenAI’s GPT Builder was used to create a GPT tailored to extract information from academic manuscripts. The Role, Instruction, Steps, End goal, and Narrowing (RISEN) framework was used to inform prompt engineering for the GPT. A sample of 20 studies from two distinct systematic reviews was used to evaluate the GPT’s performance in extraction. Agreement rates between the GPT outputs and human reviewers were calculated for each study subsection. Results: The mean time for human data extraction was 36 minutes per study, compared to 26.6 seconds for GPT generation, followed by 13 minutes of human review. The GPT demonstrated high overall agreement rates with human reviewers, achieving 91.45% for review 1 and 89.31% for review 2. It was particularly accurate in extracting study characteristics (review 1: 95.25%; review 2: 90.83%) and participant characteristics (review 1: 95.03%; review 2: 90.00%), with lower performance observed in more complex areas such as methodological characteristics (87.07%) and statistical results (77.50%). The GPT correctly extracted data in 14 instances (3.25% in review 1) and four instances (1.16% in review 2) when the human reviewer was incorrect. Conclusions: The custom GPT significantly reduced extraction time and shows evidence that it can extract data with high accuracy, particularly for participant and study characteristics. This tool may offer a viable option for researchers seeking to reduce resource demands during the extraction phase, although more research is needed to evaluate test-retest reliability, performance across broader review types, and accuracy in extracting statistical data. The tool developed in the current study has been made open access.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“a recent study showed that providing structured templates or examples of correctly formatted output can improve extraction accuracy in automated systematic review systems (17).”
From Performance of large language models and prompt engineering strategies for data extraction in systematic reviews (2026) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although these proof-of-concept studies have reported encouraging results and the potential to substantially reduce manual workload and human error, most evaluations have been limited to single models, small or restricted datasets, or single extraction runs (15–20).”
From Performance of large language models and prompt engineering strategies for data extraction in systematic reviews (2026) · cites this paper as [17]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Earlier reports have shown that LLMs can markedly reduce data extraction time compared with manual approaches (17, 21, 24).”
From Performance of large language models and prompt engineering strategies for data extraction in systematic reviews (2026) · cites this paper as [17]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“outcome data extraction, which often involves complex statistics and more nuanced contextual information (17, 22, 24), was substantially more challenging.”
From Performance of large language models and prompt engineering strategies for data extraction in systematic reviews (2026) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
The citation lists are queued
They are fetched later and appear here then.
References 22 · Cited by 16
Research works printed but not linked: 2
- Covidence systematic review software. 2025. URL: https://www.covidence.org/ [Accessed 09-09-2024]
- Systematic review extractor pro. ChatGPT. URL: https://chatgpt.com/g/g-5IRhGHEcc-systematic-review-extractor-pro [Accessed 07-08-2025]
Metadata
- Authors
- Sercombe, J., Bryant, Z., & Wilson, J.
- Affiliations
- The University of Sydney (AU)
- Corresponding author
- Bryant, Zachary
- Venue
- JMIR Formative Research
- Publisher
- JMIR Publications Inc.
- Published
- 2025-08-11
- Volume
- 9
- Pages
- e68666-e68666
- ISSN
- 2561-326X (ISSN-L)
- PMC
- PMC12338963
- OpenAlex
- W4412010526
- Semantic Scholar
- 3fcbcdffc0868e3e3e33539f8d92b9a1f8801a62
- Cited by
- 16
- References
- 22
- Citations by year
- 2026: 16
- FWCI
- 2.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Formative Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- ChatGPT, large language models, automated data extraction, systematic reviews, prompt engineering, human-AI agreement, health research
- MeSH terms
- Systematic Reviews as Topic, Generative Artificial Intelligence, Artificial Intelligence, Humans, Reproducibility of Results, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating a Customized Version of ChatGPT for Systematic Review Data Extraction in Health Research: Development and Usability Study. https://app.cochat.ai/papers/10.2196/68666
- The paper
- Sercombe, J., Bryant, Z., & Wilson, J. (2025). Evaluating a Customized Version of ChatGPT for Systematic Review Data Extraction in Health Research: Development and Usability Study. JMIR Formative Research. https://doi.org/10.2196/68666
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
