The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index; 2 of the 2 sources its claims cite were checked for support.
Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis
Open accessPublished version of a preprint
Abstract
Background Large language models (LLMs) have raised both interest and concern in the academic community. They offer the potential for automating literature search and synthesis for systematic reviews but raise concerns regarding their reliability, as the tendency to generate unsupported (hallucinated) content persist. Objective The aim of the study is to assess the performance of LLMs such as ChatGPT and Bard (subsequently rebranded Gemini) to produce references in the context of scientific writing. Methods The performance of ChatGPT and Bard in replicating the results of human-conducted systematic reviews was assessed. Using systematic reviews pertaining to shoulder rotator cuff pathology, these LLMs were tested by providing the same inclusion criteria and comparing the results with original systematic review references, serving as gold standards. The study used 3 key performance metrics: recall, precision, and F1-score, alongside the hallucination rate. Papers were considered “hallucinated” if any 2 of the following information were wrong: title, first author, or year of publication. Results In total, 11 systematic reviews across 4 fields yielded 33 prompts to LLMs (3 LLMs×11 reviews), with 471 references analyzed. Precision rates for GPT-3.5, GPT-4, and Bard were 9.4% (13/139), 13.4% (16/119), and 0% (0/104) respectively (P<.001). Recall rates were 11.9% (13/109) for GPT-3.5 and 13.7% (15/109) for GPT-4, with Bard failing to retrieve any relevant papers (P<.001). Hallucination rates stood at 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001). Further analysis of nonhallucinated papers retrieved by GPT models revealed significant differences in identifying various criteria, such as randomized studies, participant criteria, and intervention criteria. The study also noted the geographical and open-access biases in the papers retrieved by the LLMs. Conclusions Given their current performance, it is not recommended for LLMs to be deployed as the primary or exclusive tool for conducting systematic reviews. Any references generated by such models warrant thorough validation by researchers. The high occurrence of hallucinations in LLMs highlights the necessity for refining their training and functionality before confidently using them for rigorous academic purposes.
Overview
CoChat Summary
## TL;DR
This comparative analysis tested ChatGPT and Bard on 11 systematic reviews of shoulder rotator cuff pathology and found low retrieval accuracy plus frequent hallucinated references. GPT-4 performed best of the models tested, but the authors conclude that LLM-generated citations still require rigorous human validation and should not be used alone for systematic reviews.
## Key findings
- Across 11 systematic reviews in 4 fields, the study analyzed 471 references from 33 LLM prompts (3 models × 11 reviews).
- Precision was 9.4% (13/139) for GPT-3.5, 13.4% (16/119) for GPT-4, and 0% (0/104) for Bard (P<.001); recall was 11.9% (13/109) for GPT-3.5, 13.7% (15/109) for GPT-4, and 0 for Bard (P<.001).
- Hallucination rates were 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001).
- Among nonhallucinated GPT-retrieved papers, title accuracy was very high (83/84, 99% for GPT-3.5 and 84/85, 99% for GPT-4), but DOI accuracy was low (13/82, 16% and 17/84, 20%); American authors were overrepresented in ChatGPT results (44% vs 16.5% in the original reviews for GPT-3.5, and 33% vs 16.5% for GPT-4; P<.001).
## Methods
- Sequential comparative design: PubMed search on July 27, 2023 identified English-language 2020 systematic reviews/meta-analyses on randomized clinical trials in shoulder rotator cuff pathology; 11 systematic reviews were selected after title/abstract/full-text screening and validation with PRISMA/PROSPERO.
- The LLMs tested were GPT-3.5, GPT-4, and Bard (PaLM 2.0); each fresh chat session was prompted with the same inclusion criteria, and two prompt versions were tried, keeping the one that produced more results.
- Outputs were compared with the original systematic review references using recall, precision, F1-score, and hallucination rate; papers were labeled hallucinated if any 2 of title, first author, or year were wrong.
## Limitations
- The authors state the study was restricted to shoulder rotator cuff pathology and to 3 LLMs, so the findings may not generalize to other specialties or models.
- The paper also notes that no established guidelines exist for using LLMs to optimize accuracy, and different prompts might yield different results.
- Evident from the text: the study focused on a narrow, single-topic set of reviews from one year (2020) and English-language publications only, which limits breadth and may constrain generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
1 Valid · 0 Not valid · 0 Not checked · 40 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Papers from American authors were selected in 16.5% (18/109) of original systematic reviews, 44% (37/84) of GPT-3.5 papers, and 33% (28/85) of GPT-4 papers (P<.001).”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In most cases, the number of references returned by LLMs was greater or equal to that of the original papers (Table 2).”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Papers identified by LLMs were present in the original systematic reviews (precision) in 9.4% (13/139), 13.4% (16/119), and 0% (0/104) of cases for GPT-3.5, GPT-4, and Bard (P<.001), respectively.”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“authors list (73/84, 87% vs 74/85, 87%; P>.99)”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“randomized studies (33/84, 39% vs 42/85, 49%; P=.24)”
- Also stated in: § Principal Findings · plain
“For instance, the criterion of “randomized study” was adhered to in only 39% (33/84) to 49% (42/85) of nonhallucinated papers generated by ChatGPT, even when the term “randomized” appeared in the title or abstract of the papers from the original systematic reviews.”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Also stated in: § Principal Findings · plain
18 more claims
Create a free account to see all 41 claims and every source verdict.
- It may state two separate claims as one. 12
- Probably not a substantial claim. 9
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Hallucination rates have been documented to vary between 3% – 28% in current literature [30, 31].”
From Artificial Intelligence (AI) – Powered Documentation Systems in Healthcare: A Systematic Review (2025) · cites this paper as [30]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies in radiology, psychiatry, and clinical rehabilitation have reported fabrication rates ranging from 20 to 70%”
From How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations (2026) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Experimental studies from 2023 to 2025 have found that the rate of hallucination in LLMgenerated citations ranged from 19.9% to 91.4%, varying widely across models and time [16], [17], [18], [19], [24].”
From LLM hallucinations in the wild: Large-scale evidence from non-existent citations (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When citations are missing, inaccurate, or fabricated, the resulting break in the evidence chain can obscure logical dependencies, weaken argumentation, and jeopardize academic integrity at the level of both individual papers and the broader literature [10, 22, 26].”
From CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Chelli et al. found that Bard produced incorrect references in 91.4% of systematic review prompts, whereas GPT-4 had a lower but still notable error rate of 28.6%18.”
From Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support (2025) · cites this paper as [18]
Experimental — accuracy not yet measured
16 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · cited 1×
“This concern has been broached in previous literature [26-29], but our study provides an experimental design to probe the matter more deeply.”
BackgroundThe nuts and bolts of PROSPERO: an international prospective register of systematic reviews
Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-2 · Cited by 1,433 · cited 1×
“Additionally, the registration status of these reviews was verified in the PROSPERO database [14].”
BackgroundChatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355
ChatGPT listed as author on research papers: many scientists disapprove
Stokel-Walker, C. (2023) · Nature · doi:10.1038/d41586-023-00107-z · Cited by 868 · cited 1×
“While it is known that ChatGPT can help researchers write papers [3-5], controversy exists about whether it should be used at all, whether its use should be disclosed, and whether it should be listed as an author or not [6].”
BackgroundCan artificial intelligence help for scientific writing?
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04380-2 · Cited by 827
Abstracts written by ChatGPT fool scientists
Else, H. (2023) · Nature · doi:10.1038/d41586-023-00056-7 · Cited by 611
Boileau, P., Baqué, F., Valerio, L., Ahrens, P., Chuinard, C., & Trojani, C. (2007) · The Journal of Bone and Joint Surgery-American Volume · doi:10.2106/00004623-200704000-00008 · Cited by 603
Self-citationChatGPT and the future of medical writing
Biswas, S. (2023) · Radiology · doi:10.1148/radiol.223312 · Cited by 571
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
Cited by
Promises and challenges of generative artificial intelligence for human learning
Yan, L., Greiff, S., Teuber, Z., & Gašević, D. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02004-5 · Cited by 499
“GenAI may perform reasonably well with generic questions (for example, what are Newton’s laws of motion?) but is more prone to errors with nuanced, contextspecific, timesensitive or highly technical information 73 .”
Omar, M., Sorin, V., Collins, J. D., Reich, D., Freeman, R., Gavin, N., Charney, A., Stump, L., Bragazzi, N. L., Nadkarni, G. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-025-01021-3 · Cited by 204
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf063 · Cited by 154
Artificial Intelligence (AI) – Powered Documentation Systems in Healthcare: A Systematic Review
Bracken, A., Reilly, C., Feeley, A., Sheehan, E., Merghani, K., & Feeley, I. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02157-4 · Cited by 113
“Hallucination rates have been documented to vary between 3% – 28% in current literature [30, 31].”
AI hallucinations can’t be stopped — but these techniques can limit their damage
Jones, N. (2025) · Nature · doi:10.1038/d41586-025-00068-5 · Cited by 117
Metadata
- Authors
- Chelli, M., Descamps, J., Lavoué, V., Trojani, C., Azar, M., Deckert, M., Raynier, J.-L., Clowez, G., Boileau, P., & Ruetsch-Chelli, C.
- Affiliations
- Assistance Publique – Hôpitaux de Paris (FR)
- Inserm (FR)
- Corresponding author
- Chelli, Mikaël
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-05-22
- Volume
- 26
- Pages
- e53164
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11153973
- OpenAlex
- W4398203672
- Semantic Scholar
- 9a0e8c101b72e9094ea554194698daa3aeb3bc6f
- Cited by
- 602
- References
- 30
- Citations by year
- 2026: 378 · 2025: 202 · 2024: 22
- FWCI
- 11.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- hallucinations, large language models, ChatGPT, Google Bard, Gemini, systematic reviews, reference retrieval, literature search automation, precision and recall, citation bias, GPT-3.5, GPT-4, AI in scientific writing
- MeSH terms
- Systematic Reviews as Topic, Artificial Intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.