The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 16 of the 20 sources its claims cite were checked for support.
Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening?
Open accessAbstract
BACKGROUND: Literature screening constitutes a critical component in evidence synthesis; however, it typically requires substantial time and human resources. Artificial intelligence (AI) has shown promise in this field, yet the accuracy and effectiveness of AI tools for literature screening remain uncertain. This study aims to evaluate the performance of several existing AI-powered automated tools for literature screening. METHODS: This diagnostic accuracy study employed a cohort to evaluate the performance of five AI tools-ChatGPT 4.0, Claude 3.5, Gemini 1.5, DeepSeek-V3, and RobotSearch-in literature screening. We selected a random sample of 1,000 publications from a well-established literature cohort, with 500 as randomized controlled trials (RCTs) group and 500 as others group. Diagnostic accuracy was measured using several metrics, including the false negative fraction (FNF), time used for screening, false positive fraction (FPF), and the redundancy number needed to screen. RESULTS: We reported the FNF for the RCTs group and the FPF for the others group. In the RCTs group, RobotSearch exhibited the lowest FNF at 6.4% (95% CI: 4.6% to 8.9%), whereas Gemini exhibited the highest at 13.0% (95% CI: 10.3% to 16.3%). In the others group, the FPF of the four large language models ranged from 2.8% (95% CI: 1.7% to 4.7%) to 3.8% (95% CI: 2.4% to 5.9%), both of which were significantly lower than RobotSearch's rate of 22.2% (95% CI: 18.8% to 26.1%). In terms of screening efficiency, the mean time used for screening per article was 1.3 s for ChatGPT, 6.0 s for Claude, 1.2 s for Gemini, and 2.6 s for DeepSeek. CONCLUSIONS: The AI tools assessed in this study demonstrated commendable performance in literature screening; however, they are not yet suitable as standalone solutions. These tools can serve as effective auxiliary aids, and a hybrid approach that integrates human expertise with AI may enhance both the efficiency and accuracy of the literature screening process.
Overview
CoChat Summary
## TL;DR
This diagnostic accuracy study compared five AI-powered tools for screening a sample of 1,000 publications drawn from a literature cohort, split evenly between RCTs and non-RCTs. The large language models had low false positive fractions and very fast screening times, but none were accurate enough to replace human screening on their own.
## Key findings
- In the RCTs group, RobotSearch had the lowest false negative fraction (FNF) at 6.4% (95% CI: 4.6% to 8.9%), while Gemini had the highest at 13.0% (95% CI: 10.3% to 16.3%).
- In the others group, the four LLMs had false positive fractions (FPF) from 2.8% to 3.8%, all far below RobotSearch’s 22.2% (95% CI: 18.8% to 26.1%).
- Mean screening time per article was 1.3 s for ChatGPT, 6.0 s for Claude, 1.2 s for Gemini, and 2.6 s for DeepSeek; total times were 22.0 min, 100.0 min, 20.0 min, and 43.5 min, respectively.
- Redundancy numbers needed to screen were 16 for ChatGPT, 14 for Claude, 19 for Gemini, 17 for DeepSeek, and 111 for RobotSearch.
## Methods
- Diagnostic accuracy study using a cohort of literature; the paper says it followed STARD “where applicable.”
- Literature cohort: 8,394 retractions from the Retraction Watch database up to April 26, 2023; after human screening, 779 were RCTs and 7,595 were not, and the study drew a random sample of 500 from each group.
- Compared five tools: RobotSearch, ChatGPT 4.0, Claude 3.5, Gemini 1.5, and DeepSeek-V3; prompts for the four LLMs were iteratively engineered and then applied to screening.
- Main analysis focused on FNF in the RCT group and FPF in the non-RCT group, plus screening time, redundancy number needed to screen, PLR, Youden’s Index, RD, and RR.
## Limitations
- The authors state that the literature cohort was limited to retractions in the Retraction Watch database up to April 26, 2023, and did not include databases related to other disease types.
- They state that, because LLM outputs vary, they did not assess consistency of the same prompt across multiple iterations.
- They state that the approach predominantly targets RCTs and does not encompass other study types, so its effectiveness for other literature types remains to be validated.
- Evident from the text: the study assessed only a 1,000-article sample from one literature cohort, so generalizability beyond that sample is limited.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
8 Valid · 0 Not valid · 3 Not checked · 44 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Nevertheless, their performance remains inferior to that of humans [13, 16].”
§ Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? > Discussion
What the sources say
- Could not verifycites [13] Waffenschmidt S, Knelangen M, Sieben W, Buhn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol. 2019;19(1):132. sourceExperimental — accuracy not yet measured
The Methods section (Eligibility criteria and selection of evaluations; page unavailable) says that evaluations using text-mining or automation tools were excluded. The Results and Conclusions compare human single screening with human double screening, not AI tools with humans. Our copy therefore does not address the claimed performance comparison. (LLM, conf=0.78)
- Partly supportedcites [16] Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K. Can large language models replace humans in systematic reviews? Evaluating GPT-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Res Synth Methods. 2024;15(4):616–26. sourceExperimental — accuracy not yet measured
The Discussion (p. 6) says GPT-4 often substantially underperformed human reviewers after adjustment for chance agreement and dataset imbalance. But it also reports performance on par with humans under specific screening conditions, so the unqualified claim that AI tools remain inferior overstates this source's findings. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Compared to previous studies, ChatGPT in our study achieve the relatively low FNF [16, 29].”
§ Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? > Discussion
What the sources say
- Supportedcites [16] Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K. Can large language models replace humans in systematic reviews? Evaluating GPT-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Res Synth Methods. 2024;15(4):616–26. source
The citation is offered as a previous-study comparator, not as evidence for the citing authors’ own result. The cited paper defines sensitivity using true positives and false negatives (Analysis metrics, p. 4) and reports 67% sensitivity for its title/abstract screening, implying a 33% false-negative fraction for that comparison (Discussion, p. 6). (LLM, conf=0.84)
- Partly supportedcites [29] Leon M, Ruaengsri C, Pelletier G, Bethencourt D, Shibata M, Flores MQ, et al. Harnessing the power of ChatGPT in cardiovascular medicine: innovations, challenges, and future directions. J Clin Med. 2024. https://doi.org/10.3390/jcm13216543. sourceExperimental — accuracy not yet measured
Section 5 (page not available) reports low false-negative counts and 95% sensitivity for ChatGPT in a previous abstract-screening study. It does not report the citing authors’ study or compare its false-negative fraction with that earlier result, so it cannot establish the claimed relative difference. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Literature screening is the most time-consuming process in evidence synthesis practice, particularly in the medical field [9–11]”
§ Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? > Introduction
What the sources say
- Partly supportedcites [9] Lin Y, Li J, Xiao H, Zheng L, Xiao Y, Song H, et al. Automatic literature screening using the PAJO deep-learning model for clinical practice guidelines. BMC Med Inform Decis Mak. 2023;23(1):247. sourceExperimental — accuracy not yet measured
The Abstract’s Background and the Conclusions describe manual screening or curation for clinical practice guidelines as time-consuming and labor-intensive. They do not say literature screening is the most time-consuming process in evidence synthesis; the comparative claim is unsupported. Page numbers are unavailable in our copy. (LLM, conf=0.85)
- Partly supportedcites [10] Delgado-Chaves FM, Jennings MJ, Atalaia A, Wolff J, Horvath R, Mamdouh ZM, et al. Transforming literature screening: the emerging role of large language models in systematic reviews. Proc Natl Acad Sci U S A. 2025;122(2):e2411962122. sourceExperimental — accuracy not yet measured
The Abstract describes study-selection screening as notably laborious, typically taking several working days per researcher, and discusses systematic reviews used to inform clinical recommendations. It does not compare screening with other evidence-synthesis tasks or establish that screening is the most time-consuming process. (LLM, conf=0.85)
- Not checked — no full text of the sourcecites [11] Kamtchum-Tatuene J, Zafack JG. Keeping up with the medical literature: why, how, and when? Stroke. 2021;52(11):e746–8. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This may due to the limitations of traditional tools, which often segment text into fixed portions, potentially hindering their ability to capture contextual nuances, particularly in longer text [25].”
§ Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? > Discussion
What the sources say
- Partly supportedcites [25] Guo E, Gupta M, Deng J, Park YJ, Paget M, Naugler C. Automated paper screening for clinical reviews using large language models: data analysis study. J Med Internet Res. 2024;26:e48996. sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) contrasts traditional n-gram methods with GPT embeddings that represent words’ semantics and context, supporting the general point about contextual limitations. It does not say traditional tools segment text into fixed portions or that this limitation is particularly consequential for longer text. (LLM, conf=0.81)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“a significant difference in FNF was observed between Claude and Robotsearch, with a percentage difference of 4.6% (95% CI: 1.1%, 8.1%, P = 0.010)”
§ Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? > Results > Performance of automatic screen tools
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
30 more claims
Create a free account to see all 55 claims and every source verdict.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 12
Citations
Cites
Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262 · cited 1×
“A well-conducted evidence synthesis requires meticulous attention to detail at each step, making the process often time-and labor-intensive [5–7].”
Cumpston, M., Li, T., Page, M. J., Chandler, J., Welch, V. A., Higgins, J. P., & Thomas, J. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000142 · Cited by 7,218 · cited 1×
“In evidence synthesis practice, standardized procedures are crucial in safeguarding against potential biases (e.g., publication bias) and errors (e.g., human error), thereby preventing adverse effects on the validity of the summarized evidence [4].”
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“For example, a prompt such as “Please design the best prompt for me based on this prompt: …” was used in this step [26].”
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · cited 1×
“This facilitates the development of artificial intelligence for the screening process, thereby alleviating human workload and enhancing efficiency [15–17].”
Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215 · cited 1×
“The study was conducted in accordance with the STARD (Standards for Reporting Diagnostic Accuracy Studies) guidelines, where applicable [23].”
Aromataris, E., Fernandez, R., Godfrey, C. M., Holly, C., Khalil, H., & Tungpunkom, P. (2015) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000055 · Cited by 2,587 · cited 1×
“As its value has gained increasing recognition, evidence synthesis has emerged as a fundamental component of the contemporary medical paradigm, producing evidences through these syntheses and supporting clinical guidelines and healthcare policies development [3].”
Meta-analysis and the science of research synthesis
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018) · Nature · doi:10.1038/nature25753 · Cited by 1,709 · cited 1×
“Evidence synthesis, often referred to as research synthesis, is an essential methodology for aggregating evidence from existing studies to inform healthcare decision-making [1].”
CONSORT 2010 statement: extension to randomised crossover trials
Dwan, K., Li, T., Altman, D. G., & Elbourne, D. (2019) · BMJ · doi:10.1136/bmj.l4378 · Cited by 785 · cited 1×
“An RCT was defined as a study in which participants were explicitly randomized into two or more groups, each receiving different interventions, as determined from the full text of the study [22].”
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737
Research works printed but not linked: 1
- T L, JPT H, (editors) DJ. Chapter 5: Collecting data. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane; 2022.
Cited by
Artificial Intelligence Tools in Biomedical Research: Part 1—Literature Search and Knowledge Mining
Sen, C. K. (2025) · Antioxidants and Redox Signaling · doi:10.1177/15230864251405885 · Cited by 6
Mortezaagha, P., Shaw, J., Sun, B., & Rahgozar, A. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-026-02847-8 · Cited by 8
“Across a corpus focused on DOAC level measurement, the system achieved stable large-scale processing, produced internally consistent study-level records, and yielded measurable gains in extraction fidelity after iterative refinement of prompts and schemas.”
Mulla, A., Lee, D., Li, Y., & Dehghani, A. (2026) · Conference on Human Centred Artificial Intelligence - Education and Practice (HCAIep) · doi:10.1145/3777490.3777511 · Cited by 2
Mazzali, C., Pinciroli, T., Valetto, M. R., Dri, P., Russo, A. G., Air and Health Atlas Study Group, Angelini, P., Broccoli, S., Monti, M., Caranci, N., et al. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-026-02892-3 · Cited by 1
Applications of DeepSeek in Medicine: Bibliometric Analysis and Scoping Review
Zhang, H., Wang, D., Xu, Y., Han, S., & Wang, G. (2026) · Journal of Medical Internet Research · doi:10.2196/93354 · Cited by 1
Metadata
- Authors
- Ruan, M., Fan, J., Liu, M., Meng, Z., Zhang, X., & Zhang, C.
- Affiliations
- Second Military Medical University (CN)
- Shanghai Jiao Tong University (CN)
- Corresponding author
- Zhang, Xiaohai
- Zhang, Chengjing
- ORCID
- Zhang, Xiaohai
- Venue
- BMC Medical Research Methodology
- Published
- 2025-08-25
- Volume
- 25
- Issue
- 1
- Pages
- 199-199
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC12376440
- OpenAlex
- W4413499075
- Semantic Scholar
- 2c9e62eb17cb8d36a2740fb6cd5bbe17cc4a3fd1
- Cited by
- 12
- References
- 31
- Citations by year
- 2026: 11 · 2025: 1
- FWCI
- 7.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- 2024 Research Plan Project of the Jiading District Health Commission (2024-KY-JB-30)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- artificial intelligence, literature screening, evidence synthesis, large language models, ChatGPT, Claude, Gemini, DeepSeek, diagnostic accuracy, false positive rate, randomized controlled trials, screening efficiency
- MeSH terms
- Artificial Intelligence, Humans, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? https://app.cochat.ai/papers/10.1186/s12874-025-02644-9
- The paper
- Ruan, M., Fan, J., Liu, M., Meng, Z., Zhang, X., & Zhang, C. (2025). Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? BMC Medical Research Methodology. https://doi.org/10.1186/s12874-025-02644-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

