Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 16 of the 20 sources its claims cite were checked for support.

Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening?

Open access
Ruan, Minghao, Fan, Junhao, Liu, Mingkai, Meng, Zhengfeng, Zhang, Xiaohai, Zhang, Chengjing · 2025 · BMC Medical Research Methodology

Abstract

BACKGROUND: Literature screening constitutes a critical component in evidence synthesis; however, it typically requires substantial time and human resources. Artificial intelligence (AI) has shown promise in this field, yet the accuracy and effectiveness of AI tools for literature screening remain uncertain. This study aims to evaluate the performance of several existing AI-powered automated tools for literature screening. METHODS: This diagnostic accuracy study employed a cohort to evaluate the performance of five AI tools-ChatGPT 4.0, Claude 3.5, Gemini 1.5, DeepSeek-V3, and RobotSearch-in literature screening. We selected a random sample of 1,000 publications from a well-established literature cohort, with 500 as randomized controlled trials (RCTs) group and 500 as others group. Diagnostic accuracy was measured using several metrics, including the false negative fraction (FNF), time used for screening, false positive fraction (FPF), and the redundancy number needed to screen. RESULTS: We reported the FNF for the RCTs group and the FPF for the others group. In the RCTs group, RobotSearch exhibited the lowest FNF at 6.4% (95% CI: 4.6% to 8.9%), whereas Gemini exhibited the highest at 13.0% (95% CI: 10.3% to 16.3%). In the others group, the FPF of the four large language models ranged from 2.8% (95% CI: 1.7% to 4.7%) to 3.8% (95% CI: 2.4% to 5.9%), both of which were significantly lower than RobotSearch's rate of 22.2% (95% CI: 18.8% to 26.1%). In terms of screening efficiency, the mean time used for screening per article was 1.3 s for ChatGPT, 6.0 s for Claude, 1.2 s for Gemini, and 2.6 s for DeepSeek. CONCLUSIONS: The AI tools assessed in this study demonstrated commendable performance in literature screening; however, they are not yet suitable as standalone solutions. These tools can serve as effective auxiliary aids, and a hybrid approach that integrates human expertise with AI may enhance both the efficiency and accuracy of the literature screening process.

Overview

CoChat Summary

## TL;DR

This diagnostic accuracy study compared five AI-powered tools for screening a sample of 1,000 publications drawn from a literature cohort, split evenly between RCTs and non-RCTs. The large language models had low false positive fractions and very fast screening times, but none were accurate enough to replace human screening on their own.

## Key findings

  • In the RCTs group, RobotSearch had the lowest false negative fraction (FNF) at 6.4% (95% CI: 4.6% to 8.9%), while Gemini had the highest at 13.0% (95% CI: 10.3% to 16.3%).
  • In the others group, the four LLMs had false positive fractions (FPF) from 2.8% to 3.8%, all far below RobotSearch’s 22.2% (95% CI: 18.8% to 26.1%).
  • Mean screening time per article was 1.3 s for ChatGPT, 6.0 s for Claude, 1.2 s for Gemini, and 2.6 s for DeepSeek; total times were 22.0 min, 100.0 min, 20.0 min, and 43.5 min, respectively.
  • Redundancy numbers needed to screen were 16 for ChatGPT, 14 for Claude, 19 for Gemini, 17 for DeepSeek, and 111 for RobotSearch.

## Methods

  • Diagnostic accuracy study using a cohort of literature; the paper says it followed STARD “where applicable.”
  • Literature cohort: 8,394 retractions from the Retraction Watch database up to April 26, 2023; after human screening, 779 were RCTs and 7,595 were not, and the study drew a random sample of 500 from each group.
  • Compared five tools: RobotSearch, ChatGPT 4.0, Claude 3.5, Gemini 1.5, and DeepSeek-V3; prompts for the four LLMs were iteratively engineered and then applied to screening.
  • Main analysis focused on FNF in the RCT group and FPF in the non-RCT group, plus screening time, redundancy number needed to screen, PLR, Youden’s Index, RD, and RR.

## Limitations

  • The authors state that the literature cohort was limited to retractions in the Retraction Watch database up to April 26, 2023, and did not include databases related to other disease types.
  • They state that, because LLM outputs vary, they did not assess consistency of the same prompt across multiple iterations.
  • They state that the approach predominantly targets RCTs and does not encompass other study types, so its effectiveness for other literature types remains to be validated.
  • Evident from the text: the study assessed only a 1,000-article sample from one literature cohort, so generalizability beyond that sample is limited.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s12874-025-02644-9

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Performance of the AI tools for literature screening
Fig. 1 Performance of the AI tools for literature screeningCC BY · PubMed Central
Fig. 2 Percentage difference in the performance of the AI tools
Fig. 2 Percentage difference in the performance of the AI toolsCC BY · PubMed Central

Verification

Claims the paper makes

8 Valid · 0 Not valid · 3 Not checked · 44 Own results and interpretations

30 more claims

Create a free account to see all 55 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 8
  • It may state two separate claims as one. 12

Create a free account to see these claims.

Citations

Cites

Show all 31 references

Research works printed but not linked: 1

  1. T L, JPT H, (editors) DJ. Chapter 5: Collecting data. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane; 2022.

Cited by

See all 12

Metadata

Authors
Ruan, M., Fan, J., Liu, M., Meng, Z., Zhang, X., & Zhang, C.
Affiliations
  • Second Military Medical University (CN)
  • Shanghai Jiao Tong University (CN)
Corresponding author
  • Zhang, Xiaohai
  • Zhang, Chengjing
Venue
BMC Medical Research Methodology
Published
2025-08-25
Volume
25
Issue
1
Pages
199-199
ISSN
1471-2288 (ISSN-L)
OpenAlex
W4413499075
Cited by
12
References
31
Citations by year
2026: 11 · 2025: 1
FWCI
7.61
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
2024 Research Plan Project of the Jiading District Health Commission (2024-KY-JB-30)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
artificial intelligence, literature screening, evidence synthesis, large language models, ChatGPT, Claude, Gemini, DeepSeek, diagnostic accuracy, false positive rate, randomized controlled trials, screening efficiency
MeSH terms
Artificial Intelligence, Humans, Randomized Controlled Trials as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? https://app.cochat.ai/papers/10.1186/s12874-025-02644-9
The paper
Ruan, M., Fan, J., Liu, M., Meng, Z., Zhang, X., & Zhang, C. (2025). Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? BMC Medical Research Methodology. https://doi.org/10.1186/s12874-025-02644-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.