CoChat
Cited by 201

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation?

Open access
Qureshi, Riaz, Shaughnessy, Daniel, Gill, Kayden A. R., Robinson, Karen A., Li, Tianjing, Agai, Eitan · 2023 · Systematic Reviews

Abstract

In this commentary, we discuss ChatGPT and our perspectives on its utility to systematic reviews (SRs) through the appropriateness and applicability of its responses to SR related prompts. The advancement of artificial intelligence (AI)-assisted technologies leave many wondering about the current capabilities, limitations, and opportunities for integration AI into scientific endeavors. Large language models (LLM)—such as ChatGPT, designed by OpenAI—have recently gained widespread attention with their ability to respond to various prompts in a natural-sounding way. Systematic reviews (SRs) utilize secondary data and often require many months and substantial financial resources to complete, making them attractive grounds for developing AI-assistive technologies. On February 6, 2023, PICO Portal developers hosted a webinar to explore ChatGPT’s responses to tasks related to SR methodology. Our experience from exploring the responses of ChatGPT suggest that while ChatGPT and LLMs show some promise for aiding in SR-related tasks, the technology is in its infancy and needs much development for such applications. Furthermore, we advise that great caution should be taken by non-content experts in using these tools due to much of the output appearing, at a high level, to be valid, while much is erroneous and in need of active vetting.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13643-023-02243-z

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 7 references

Research works printed but not linked: 4

  1. Mahesh B. Machine learning algorithms - a review. Int J Sci Res. 2018;18(8):381–6. https://doi.org/10.21275/ART20203995. · DOI · Paper page
  2. Covidence. Covidence - better systematic review management. Covidence. Published 2023. Accessed 10 Feb 2023. https://www.covidence.org/.
  3. Rayyan. Rayyan - Intelligent Systematic Review. Faster Systematic Reviews. Published 2023. www.rayyan.ai.
  4. Lewis P, Ott M, Du J, Stoyanov V. Pretrained language models for biomedical and clinical tasks: understanding and extending the state-of-the-art. In: Proceedings Of the 3rd Clinical Natural Language Processing Workshop. 2020. p. 146–57. https://doi.org/10.18653/v1/2020.clinicalnlp-1.17. · DOI · Paper page

Metadata

Authors
Qureshi, R., Shaughnessy, D., Gill, K. A. R., Robinson, K. A., Li, T., & Agai, E.
Venue
Systematic Reviews
Published
2023-04-29
Volume
12
Issue
1
Pages
72-72
ISSN
2046-4053 (ISSN-L)
OpenAlex
W4367368990
Cited by
201
References
7
FWCI
15.40
Type
journal article
Language
en
Open access
gold
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation? https://app.cochat.ai/papers/10.1186/s13643-023-02243-z
The paper
Qureshi, R., Shaughnessy, D., Gill, K. A. R., Robinson, K. A., Li, T., & Agai, E. (2023). Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation? Systematic Reviews. https://doi.org/10.1186/s13643-023-02243-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.