Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index; 4 of the 8 sources its claims cite were checked for support.

Accelerating the pace and accuracy of systematic reviews using AI: a validation study

Open access

Abstract

BACKGROUND: Artificial intelligence (AI) can greatly enhance efficiency in systematic literature reviews and meta-analyses, but its accuracy in screening titles/abstracts and full-text articles is uncertain. OBJECTIVES: This study evaluated the performance metrics (sensitivity, specificity) of a GPT-4 AI program, Review Copilot, against human decisions (gold standard) in screening titles/abstracts and full-text articles from four published systematic reviews/meta-analyses. RESEARCH DESIGN: Participant data from four already-published systematic literature reviews were used for this validation study. This was a study comparing Review Copilot to human decision-making (gold standard) in screening titles/abstracts and full-text articles for systematic reviews/meta-analyses. The four studies that were used in this study included observational studies and randomized control trials. Review Copilot operates on the OpenAI, GPT-4 server. We examined the performance metrics of Review Copilot to include and exclude titles/abstracts and full-text articles as compared to human decisions in four systematic reviews/meta-analyses. Sensitivity, specificity, and balanced accuracy of title/abstract and full-text screening were compared between Review Copilot and human decisions. RESULTS: Review Copilot's sensitivity and specificity for title/abstract screening were 99.2% and 83.6%, respectively, and 97.6% and 47.4% for full-text screening. The average agreement between two runs was 95.4%, with a kappa statistic of 0.83. Review Copilot screened in one-quarter of the time compared to humans. CONCLUSIONS: AI use in systematic reviews and meta-analyses is inevitable. Health researchers must understand these technologies' strengths and limitations to ethically leverage them for research efficiency and evidence-based decision-making in health.

Overview

CoChat Summary

## TL;DR

This validation study compared a GPT-4 tool, Review Copilot, against human decisions in four published systematic reviews/meta-analyses. Review Copilot was highly sensitive but only moderately specific, and it screened in about one-quarter of the time required by humans.

## Key findings

  • For title/abstract screening across all four datasets, Review Copilot had 99.2% sensitivity, 83.6% specificity, and 91.4% balanced accuracy.
  • For full-text screening across all four datasets, it had 97.6% sensitivity, 47.4% specificity, and 72.5% balanced accuracy.
  • The average agreement between two AI runs was 95.4% for title/abstract screening with kappa = 0.83; for full-text screening, agreement was 89.8% with kappa = 0.74.
  • Estimated combined screening time for Review Copilot was 3.5 h, 6.5 h, 13 h, and 4.5 h for the four datasets, and the authors state it screened in one-quarter of the time compared to humans.

## Methods

  • Validation study comparing Review Copilot (a GPT-4-based pipeline) with human decisions as the gold standard for title/abstract and full-text screening.
  • Used four already-published systematic reviews/meta-analyses as datasets: two on nutrition-related interventions, one on prediabetes interventions, and one on cancer drug-transition outcomes; the reviews included observational studies and randomized controlled trials.
  • Main analyses reported sensitivity, specificity, balanced accuracy, and inter-rater reliability across two AI runs; performance was assessed using confusion matrices and kappa statistics.

## Limitations

  • The paper states the main limitation was the small number of reviews used (N = 4), because detailed human screening decisions were not readily available for more studies.
  • The validation covered intervention studies only, so the usefulness of AI for other study types, including implementation/process studies and non-health studies, was not tested.
  • Evident from the text: the datasets were all from published reviews with available detailed human decisions, so the findings may not generalize to settings without such curated gold-standard data.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13643-025-02997-8

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Confusion matrix for all datasets. Figure legend: This figure depicts the concordance and non-concordance between the gold-standard, human decision for abstract/title and full-text inclusion/exclusion (x-axis) compared to the AI/Review Copilot decision (y-axis). For example, the AI included 162 articles that humans did, yet excluded 2 articles that humans included
Fig. 1 Confusion matrix for all datasets. Figure legend: This figure depicts the concordance and non-concordance between the gold-standard, human decision for abstract/title and full-text inclusion/exclusion (x-axis) compared to the AI/Review Copilot decision (y-axis). For example, the AI included 162 articles that humans did, yet excluded 2 articles that humans includedCC BY · PubMed Central

Verification

Claims the paper makes

2 Valid · 1 Not valid · 2 Not checked · 42 Own results and interpretations

23 more claims

Create a free account to see all 47 claims and every source verdict.

See all claims — free account
  • The quote only partly states this claim. 1
  • It may state two separate claims as one. 8
  • Probably not a substantial claim. 14

Create a free account to see these claims.

Citations

Cites

Show all 21 references

Research works printed but not linked: 4

  1. Uman LS. Systematic reviews and meta-analyses. J Can Acad Child Adolesc Psychiatry. 2011;20(1):57–9.
  2. Using machine learning to advance synthesis and use of conservation and environmental evidence - PubMed. Accessed 18 Dec 2023. https://pubmed.ncbi.nlm.nih.gov/29644722/.
  3. Covidence - Better systematic review management. Covidence. Accessed 17 Jan 2024. https://www.covidence.org/.
  4. DistillerSR | Systematic Review Software | Literature Review Software. DistillerSR. Accessed 17 Jan 2024. https://www.distillersr.com/products/distillersr-systematic-review-software.

Cited by

See all 12

Metadata

Authors
Zhan, J., Suvada, K., Xu, M., Tian, W., Cara, K. C., Wallace, T. C., & Ali, M. K.
Affiliations
  • Emory University (US)
  • University at Buffalo, State University of New York (US)
  • WuXi AppTec (United States) (US)
  • American Cancer Society (US)
Corresponding author
  • Zhan, Jiada
  • Suvada, Kara
Venue
Systematic Reviews
Published
2025-12-18
Volume
15
Issue
1
Pages
24-24
ISSN
2046-4053 (ISSN-L)
OpenAlex
W4417455010
Cited by
12
References
21
Citations by year
2026: 12
FWCI
2.27
Type
journal article
Language
en
Open access
Licence
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
artificial intelligence, GPT-4, systematic reviews, meta-analysis, article screening, title and abstract screening, full-text screening, sensitivity and specificity, AI validation, research efficiency
MeSH terms
Systematic Reviews as Topic, Artificial Intelligence, Decision Making, Humans, Sensitivity and Specificity, Meta-Analysis as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Accelerating the pace and accuracy of systematic reviews using AI: a validation study. https://app.cochat.ai/papers/10.1186/s13643-025-02997-8
The paper
Zhan, J., Suvada, K., Xu, M., Tian, W., Cara, K. C., Wallace, T. C., & Ali, M. K. (2025). Accelerating the pace and accuracy of systematic reviews using AI: a validation study. Systematic Reviews. https://doi.org/10.1186/s13643-025-02997-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.