The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index; 4 of the 8 sources its claims cite were checked for support.
Accelerating the pace and accuracy of systematic reviews using AI: a validation study
Open accessAbstract
BACKGROUND: Artificial intelligence (AI) can greatly enhance efficiency in systematic literature reviews and meta-analyses, but its accuracy in screening titles/abstracts and full-text articles is uncertain. OBJECTIVES: This study evaluated the performance metrics (sensitivity, specificity) of a GPT-4 AI program, Review Copilot, against human decisions (gold standard) in screening titles/abstracts and full-text articles from four published systematic reviews/meta-analyses. RESEARCH DESIGN: Participant data from four already-published systematic literature reviews were used for this validation study. This was a study comparing Review Copilot to human decision-making (gold standard) in screening titles/abstracts and full-text articles for systematic reviews/meta-analyses. The four studies that were used in this study included observational studies and randomized control trials. Review Copilot operates on the OpenAI, GPT-4 server. We examined the performance metrics of Review Copilot to include and exclude titles/abstracts and full-text articles as compared to human decisions in four systematic reviews/meta-analyses. Sensitivity, specificity, and balanced accuracy of title/abstract and full-text screening were compared between Review Copilot and human decisions. RESULTS: Review Copilot's sensitivity and specificity for title/abstract screening were 99.2% and 83.6%, respectively, and 97.6% and 47.4% for full-text screening. The average agreement between two runs was 95.4%, with a kappa statistic of 0.83. Review Copilot screened in one-quarter of the time compared to humans. CONCLUSIONS: AI use in systematic reviews and meta-analyses is inevitable. Health researchers must understand these technologies' strengths and limitations to ethically leverage them for research efficiency and evidence-based decision-making in health.
Overview
CoChat Summary
## TL;DR
This validation study compared a GPT-4 tool, Review Copilot, against human decisions in four published systematic reviews/meta-analyses. Review Copilot was highly sensitive but only moderately specific, and it screened in about one-quarter of the time required by humans.
## Key findings
- For title/abstract screening across all four datasets, Review Copilot had 99.2% sensitivity, 83.6% specificity, and 91.4% balanced accuracy.
- For full-text screening across all four datasets, it had 97.6% sensitivity, 47.4% specificity, and 72.5% balanced accuracy.
- The average agreement between two AI runs was 95.4% for title/abstract screening with kappa = 0.83; for full-text screening, agreement was 89.8% with kappa = 0.74.
- Estimated combined screening time for Review Copilot was 3.5 h, 6.5 h, 13 h, and 4.5 h for the four datasets, and the authors state it screened in one-quarter of the time compared to humans.
## Methods
- Validation study comparing Review Copilot (a GPT-4-based pipeline) with human decisions as the gold standard for title/abstract and full-text screening.
- Used four already-published systematic reviews/meta-analyses as datasets: two on nutrition-related interventions, one on prediabetes interventions, and one on cancer drug-transition outcomes; the reviews included observational studies and randomized controlled trials.
- Main analyses reported sensitivity, specificity, balanced accuracy, and inter-rater reliability across two AI runs; performance was assessed using confusion matrices and kappa statistics.
## Limitations
- The paper states the main limitation was the small number of reviews used (N = 4), because detailed human screening decisions were not readily available for more studies.
- The validation covered intervention studies only, so the usefulness of AI for other study types, including implementation/process studies and non-health studies, was not tested.
- Evident from the text: the datasets were all from published reviews with available detailed human decisions, so the findings may not generalize to settings without such curated gold-standard data.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
2 Valid · 1 Not valid · 2 Not checked · 42 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“validation is mostly lacking in details in many machine learning tools for systematic reviews [7].”
§ Accelerating the pace and accuracy of systematic reviews using AI: a validation study > Background
What the sources say
- Partly supportedcites [7] van de Schoot R, de Bruin J, Schram R, et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell. 2021;3(2):125–33. https://doi.org/10.1038/s42256-020-00287-7. sourceExperimental — accuracy not yet measured
In the passage labelled “Not available” (page not specified), the paper says benchmark-testing options for machine-learning-aided review tools are mostly lacking. The conclusion (page not specified) also says estimating a screening system’s error rate remains difficult. These support a concern about evaluating the tools, but neither passage says that many tools provide insufficient details about their validation. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“we used human researcher-extracted data as the gold standard from four systematic reviews or meta-analyses that had already been published [16–19].”
§ Accelerating the pace and accuracy of systematic reviews using AI: a validation study > Methods and measures > Validation and performance metrics of Review Copilot
What the sources say
- Supportedcites [16] Kc C, Ar B, Tc W, M C. Safety of Using Enteral Nutrition Formulations Containing Dietary Fiber in Hospitalized Critical Care Patients: A Systematic Review and Meta-Analysis. JPEN J Parenter Enteral Nutr. 2021;45(5). https://doi.org/10.1002/jpen.2210. source
The Methods and Data extraction sections identify this paper as a systematic review and state that an investigator extracted its data. The citing authors’ use of those data as a gold standard, and their selection of four published reviews, describe their own study rather than a procedure attributed to this source. (LLM, conf=0.88)
- Supportedcites [17] Cara KC, Beauchesne AR, Wallace TC, Chung M. Effects of 100% orange juice on markers of inflammation and oxidation in healthy and at-risk adult populations: a scoping review, systematic review, and meta-analysis. Adv Nutr. 2021;13(1):116–37. https://doi.org/10.1093/advances/nmab101. source
This citation identifies one of the published reviews whose extracted data the citing authors used; it does not attribute the four-review count or the gold-standard decision to this source. The Methods (Data extraction) describe data extracted by one investigator and checked by another, and the Methods (Strength of evidence and meta-analyses) identify the paper as a systematic review with meta-analyses. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [18] Galaviz KI, Weber MB, Suvada K, et al. Interventions for reversing prediabetes: a systematic review and meta-analysis. Am J Prev Med. 2022;62(4):614–25. https://doi.org/10.1016/j.amepre.2021.10.020. source
- Supportedcites [19] Meijboom RW, Gardarsdottir H, Egberts TCG, Giezen TJ. Patients retransitioning from biosimilar TNFα inhibitor to the corresponding originator after initial transitioning to the biosimilar: a systematic review. BioDrugs. 2022;36(1):27–39. https://doi.org/10.1007/s40259-021-00508-4. source
The Results section identifies this paper as a systematic review of 37 studies and says it presents data extracted from those studies. The Discussion says the authors extracted factors from the included studies and that an independent second reviewer cross-checked the extraction. This supports citing the paper as one published review providing researcher-extracted data; it does not independently establish how the citing study used those data. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As Review Copilot can exclude about 90–95% of irrelevant articles during the title and abstract screening stage with the speed of 1000 articles per four person hours”
§ Accelerating the pace and accuracy of systematic reviews using AI: a validation study > Discussion
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the average percent agreement between the two Review Copilot’s runs was 95.4% and the average kappa statistic was 0.83.”
§ Accelerating the pace and accuracy of systematic reviews using AI: a validation study > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There were 162 true positives, 992 false positives, 2 false negatives, and 5028 true negatives (Fig. 1).”
§ Accelerating the pace and accuracy of systematic reviews using AI: a validation study > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
23 more claims
Create a free account to see all 47 claims and every source verdict.
- The quote only partly states this claim. 1
- It may state two separate claims as one. 8
- Probably not a substantial claim. 14
Citations
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092 · cited 1×
“There are many existing active learning tools, including Abstrackr [6], ASReview [7], Colandr [8], FASTREAD [9], Rayyan [10], RobotAnalyst [11], Covi-dence [12], and DistillerSR [13].”
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877 · cited 1×
“We evaluated the inter-rater reliability of two AI runs by percent agreement and kappa statistic [20].”
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · Epidemiology · doi:10.1097/ede.0b013e3181577654 · Cited by 1,991
Bornmann, L., & Mutz, R. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23329 · Cited by 1,576
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157 · cited 2×
“Second, validation is mostly lacking in details in many machine learning tools for systematic reviews [7].”
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638 · cited 1×
“The challenge is compounded by the rapid growth of literature in many fields that are important to the topic, making a comprehensive, manual review increasingly unfeasible [5].”
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532
Artificial intelligence in systematic reviews: promising when appropriately used
van Dijk, S. H. B., Brusse-Keizer, M. G. J., Bucsán, C. C., van der Palen, J., Doggen, C. J. M., & Lenferink, A. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-072254 · Cited by 256 · cited 4×
“Leveraging this technology can help generate evidence more efficiently, while simultaneously acknowledging the importance of human interpretation and application of results to health [27].”
InfluentialUsing artificial intelligence methods for systematic review in health sciences: A systematic review
Blaizot, A., Veettil, S. K., Saidoung, P., Moreno‐Garcia, C. F., Wiratunga, N., Aceves‐Martins, M., Lai, N. M., & Chaiyakunapruk, N. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1553 · Cited by 223
Alshami, A., Elsayed, M., Ali, E., Eltoukhy, A. E. E., & Zayed, T. (2023) · Systems · doi:10.3390/systems11070351 · Cited by 202 · cited 2×
“None of these articles found that AI can fully automate the review process; however, citing pitfalls such as limited ability to extract full-text articles in real-time, researcher influence/bias on the algorithm used for article review, and/or uncharted ethical considerations [21, 27].”
Research works printed but not linked: 4
- Uman LS. Systematic reviews and meta-analyses. J Can Acad Child Adolesc Psychiatry. 2011;20(1):57–9.
- Using machine learning to advance synthesis and use of conservation and environmental evidence - PubMed. Accessed 18 Dec 2023. https://pubmed.ncbi.nlm.nih.gov/29644722/.
- Covidence - Better systematic review management. Covidence. Accessed 17 Jan 2024. https://www.covidence.org/.
- DistillerSR | Systematic Review Software | Literature Review Software. DistillerSR. Accessed 17 Jan 2024. https://www.distillersr.com/products/distillersr-systematic-review-software.
Cited by
Fleurence, R., Qureshi, R., Aggarwal, R., Bian, J., Cheyne, S., Dawoud, D., Faden, R., & Chhatwal, J. (2026) · Value in Health · doi:10.1016/j.jval.2026.06.012 · Cited by 7
Bloudek, L. M., Cichewicz, A. B., Patel, K., Sullivan, S. D., & Kallmes, K. M. (2026) · PharmacoEconomics · doi:10.1007/s40273-026-01648-7
Audax: A Principled Method for AI-Assisted Evidence SynthesisPreprint
Lascher, S. (2026) · Research Square · doi:10.21203/rs.3.rs-9590065/v1
Dhurandhar, D., Trivedi, S., & Chandrakar, T. (2026) · National Journal of the Society of Medical Anatomists · doi:10.4103/njsoma.njsoma_19_26
Enhancing Title and Abstract Priority Screening Through EdRoSi AI PipelinePreprint
Lecot, P., Tonoli-Catez, H., Noseda, A., Buisse, T., Chanut, S., & Chanel, I. (2026) · medRxiv · doi:10.64898/2026.06.26.26356718
Metadata
- Authors
- Zhan, J., Suvada, K., Xu, M., Tian, W., Cara, K. C., Wallace, T. C., & Ali, M. K.
- Affiliations
- Emory University (US)
- University at Buffalo, State University of New York (US)
- WuXi AppTec (United States) (US)
- American Cancer Society (US)
- Corresponding author
- Zhan, Jiada
- Suvada, Kara
- Venue
- Systematic Reviews
- Published
- 2025-12-18
- Volume
- 15
- Issue
- 1
- Pages
- 24-24
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC12829171
- OpenAlex
- W4417455010
- Semantic Scholar
- 653c0acc5faa67236108f6f627a0bddd68d9db23
- Cited by
- 12
- References
- 21
- Citations by year
- 2026: 12
- FWCI
- 2.27
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor, 36 d)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- artificial intelligence, GPT-4, systematic reviews, meta-analysis, article screening, title and abstract screening, full-text screening, sensitivity and specificity, AI validation, research efficiency
- MeSH terms
- Systematic Reviews as Topic, Artificial Intelligence, Decision Making, Humans, Sensitivity and Specificity, Meta-Analysis as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Accelerating the pace and accuracy of systematic reviews using AI: a validation study. https://app.cochat.ai/papers/10.1186/s13643-025-02997-8
- The paper
- Zhan, J., Suvada, K., Xu, M., Tian, W., Cara, K. C., Wallace, T. C., & Ali, M. K. (2025). Accelerating the pace and accuracy of systematic reviews using AI: a validation study. Systematic Reviews. https://doi.org/10.1186/s13643-025-02997-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.