CoChat

The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index; 2 of the 2 sources its claims cite were checked for support.

Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis

Open access

Published version of a preprint

Abstract

Background Large language models (LLMs) have raised both interest and concern in the academic community. They offer the potential for automating literature search and synthesis for systematic reviews but raise concerns regarding their reliability, as the tendency to generate unsupported (hallucinated) content persist. Objective The aim of the study is to assess the performance of LLMs such as ChatGPT and Bard (subsequently rebranded Gemini) to produce references in the context of scientific writing. Methods The performance of ChatGPT and Bard in replicating the results of human-conducted systematic reviews was assessed. Using systematic reviews pertaining to shoulder rotator cuff pathology, these LLMs were tested by providing the same inclusion criteria and comparing the results with original systematic review references, serving as gold standards. The study used 3 key performance metrics: recall, precision, and F1-score, alongside the hallucination rate. Papers were considered “hallucinated” if any 2 of the following information were wrong: title, first author, or year of publication. Results In total, 11 systematic reviews across 4 fields yielded 33 prompts to LLMs (3 LLMs×11 reviews), with 471 references analyzed. Precision rates for GPT-3.5, GPT-4, and Bard were 9.4% (13/139), 13.4% (16/119), and 0% (0/104) respectively (P<.001). Recall rates were 11.9% (13/109) for GPT-3.5 and 13.7% (15/109) for GPT-4, with Bard failing to retrieve any relevant papers (P<.001). Hallucination rates stood at 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001). Further analysis of nonhallucinated papers retrieved by GPT models revealed significant differences in identifying various criteria, such as randomized studies, participant criteria, and intervention criteria. The study also noted the geographical and open-access biases in the papers retrieved by the LLMs. Conclusions Given their current performance, it is not recommended for LLMs to be deployed as the primary or exclusive tool for conducting systematic reviews. Any references generated by such models warrant thorough validation by researchers. The high occurrence of hallucinations in LLMs highlights the necessity for refining their training and functionality before confidently using them for rigorous academic purposes.

Overview

CoChat Summary

## TL;DR

This comparative analysis tested ChatGPT and Bard on 11 systematic reviews of shoulder rotator cuff pathology and found low retrieval accuracy plus frequent hallucinated references. GPT-4 performed best of the models tested, but the authors conclude that LLM-generated citations still require rigorous human validation and should not be used alone for systematic reviews.

## Key findings

  • Across 11 systematic reviews in 4 fields, the study analyzed 471 references from 33 LLM prompts (3 models × 11 reviews).
  • Precision was 9.4% (13/139) for GPT-3.5, 13.4% (16/119) for GPT-4, and 0% (0/104) for Bard (P<.001); recall was 11.9% (13/109) for GPT-3.5, 13.7% (15/109) for GPT-4, and 0 for Bard (P<.001).
  • Hallucination rates were 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001).
  • Among nonhallucinated GPT-retrieved papers, title accuracy was very high (83/84, 99% for GPT-3.5 and 84/85, 99% for GPT-4), but DOI accuracy was low (13/82, 16% and 17/84, 20%); American authors were overrepresented in ChatGPT results (44% vs 16.5% in the original reviews for GPT-3.5, and 33% vs 16.5% for GPT-4; P<.001).

## Methods

  • Sequential comparative design: PubMed search on July 27, 2023 identified English-language 2020 systematic reviews/meta-analyses on randomized clinical trials in shoulder rotator cuff pathology; 11 systematic reviews were selected after title/abstract/full-text screening and validation with PRISMA/PROSPERO.
  • The LLMs tested were GPT-3.5, GPT-4, and Bard (PaLM 2.0); each fresh chat session was prompted with the same inclusion criteria, and two prompt versions were tried, keeping the one that produced more results.
  • Outputs were compared with the original systematic review references using recall, precision, F1-score, and hallucination rate; papers were labeled hallucinated if any 2 of title, first author, or year were wrong.

## Limitations

  • The authors state the study was restricted to shoulder rotator cuff pathology and to 3 LLMs, so the findings may not generalize to other specialties or models.
  • The paper also notes that no established guidelines exist for using LLMs to optimize accuracy, and different prompts might yield different results.
  • Evident from the text: the study focused on a narrow, single-topic set of reviews from one year (2020) and English-language publications only, which limits breadth and may constrain generalizability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/53164

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Captured screenshots demonstrating a prompt to large language models.
Figure 1 Captured screenshots demonstrating a prompt to large language models.CC BY · PubMed Central
Figure 2 Flow diagram of included systematic reviews.
Figure 2 Flow diagram of included systematic reviews.CC BY · PubMed Central
Figure 3 Efficiency of the tested large language models in complying to inclusion and exclusion criteria. With the exception of the “paper exists” criteria, hallucinated papers were excluded from this analysis.
Figure 3 Efficiency of the tested large language models in complying to inclusion and exclusion criteria. With the exception of the “paper exists” criteria, hallucinated papers were excluded from this analysis.CC BY · PubMed Central
Figure 4 Efficiency of the tested large language models in generating accurate bibliographic information of the retrieved papers. Hallucinated papers were excluded from this analysis.
Figure 4 Efficiency of the tested large language models in generating accurate bibliographic information of the retrieved papers. Hallucinated papers were excluded from this analysis.CC BY · PubMed Central
Figure 5 Instances of hallucinated papers. In total, 4 of 5 paper titles commence with “Tranexamic acid for the prevention of bleeding in arthroscopic” and were allegedly published in the journal Arthroscopy.
Figure 5 Instances of hallucinated papers. In total, 4 of 5 paper titles commence with “Tranexamic acid for the prevention of bleeding in arthroscopic” and were allegedly published in the journal Arthroscopy.CC BY · PubMed Central
Figure 6 Instance of a hallucinated reference. (A) The output of a large language model. (B and C) Authentic papers with similarities in title and author list, potentially serving as original data for large language model reference generation.
Figure 6 Instance of a hallucinated reference. (A) The output of a large language model. (B and C) Authentic papers with similarities in title and author list, potentially serving as original data for large language model reference generation.CC BY · PubMed Central

Verification

Claims the paper makes

1 Valid · 0 Not valid · 0 Not checked · 40 Own results and interpretations

18 more claims

Create a free account to see all 41 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 12
  • Probably not a substantial claim. 9

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

16 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 35 references

Cited by

See all 602

Metadata

Authors
Chelli, M., Descamps, J., Lavoué, V., Trojani, C., Azar, M., Deckert, M., Raynier, J.-L., Clowez, G., Boileau, P., & Ruetsch-Chelli, C.
Affiliations
  • Assistance Publique – Hôpitaux de Paris (FR)
  • Inserm (FR)
Corresponding author
Chelli, Mikaël
Venue
Journal of Medical Internet Research
Published
2024-05-22
Volume
26
Pages
e53164
ISSN
1438-8871 (ISSN-L)
OpenAlex
W4398203672
Cited by
602
References
30
Citations by year
2026: 378 · 2025: 202 · 2024: 22
FWCI
11.51
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • HAL (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
hallucinations, large language models, ChatGPT, Google Bard, Gemini, systematic reviews, reference retrieval, literature search automation, precision and recall, citation bias, GPT-3.5, GPT-4, AI in scientific writing
MeSH terms
Systematic Reviews as Topic, Artificial Intelligence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.