CoChat

Reference Hallucination Score for Medical Artificial Intelligence Chatbots: Development and Usability Study

Open access

Published version of a preprint

The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index; 27 of the 33 sources its claims cite were checked for support.

Abstract

Background: Artificial intelligence (AI) chatbots have recently gained use in medical practice by health care practitioners. Interestingly, the output of these AI chatbots was found to have varying degrees of hallucination in content and references. Such hallucinations generate doubts about their output and their implementation. Objective: The aim of our study was to propose a reference hallucination score (RHS) to evaluate the authenticity of AI chatbots’ citations. Methods: Six AI chatbots were challenged with the same 10 medical prompts, requesting 10 references per prompt. The RHS is composed of 6 bibliographic items and the reference’s relevance to prompts’ keywords. RHS was calculated for each reference, prompt, and type of prompt (basic vs complex). The average RHS was calculated for each AI chatbot and compared across the different types of prompts and AI chatbots. Results: Bard failed to generate any references. ChatGPT 3.5 and Bing generated the highest RHS (score=11), while Elicit and SciSpace generated the lowest RHS (score=1), and Perplexity generated a middle RHS (score=7). The highest degree of hallucination was observed for reference relevancy to the prompt keywords (308/500, 61.6%), while the lowest was for reference titles (169/500, 33.8%). ChatGPT and Bing had comparable RHS (β coefficient=–0.069; P=.32), while Perplexity had significantly lower RHS than ChatGPT (β coefficient=–0.345; P<.001). AI chatbots generally had significantly higher RHS when prompted with scenarios or complex format prompts (β coefficient=0.486; P<.001). Conclusions: The variation in RHS underscores the necessity for a robust reference evaluation tool to improve the authenticity of AI chatbots. Further, the variations highlight the importance of verifying their output and citations. Elicit and SciSpace had negligible hallucination, while ChatGPT and Bing had critical hallucination levels. The proposed AI chatbots’ RHS could contribute to ongoing efforts to enhance AI’s general reliability in medical research.

Overview

CoChat Summary

TL;DR

The study proposes a Reference Hallucination Score (RHS) to evaluate the authenticity of AI chatbot citations, scoring 7 bibliographic/relevance items with a maximum score of 11. Across 6 chatbots and 500 requested references, Elicit and SciSpace had negligible hallucination, while ChatGPT 3.5 and Bing had critical hallucination and Bard generated no references.

Key findings

  • Bard failed to generate any references for all 10 prompts; overall, 35/500 (7%) references were not generated. The highest hallucination was reference relevance to prompt keywords (308/500, 61.6%), while the lowest was reference title (169/500, 33.8%); publication date was 237/500 (47.4%), authors’ names 228/500 (45.6%), DOI 227/500 (45.4%), web link 187/500 (37.4%), and journal name 188/500 (37.6%).
  • Median total RHS was highest for ChatGPT 3.5 and Bing (score=11) and lowest for Elicit and SciSpace (score=1), with Perplexity in the middle (score=7).
  • In multivariable regression, ChatGPT and Bing had comparable RHS (β coefficient=–0.069; P=.32), while Perplexity had significantly lower RHS than ChatGPT (β coefficient=–0.345; P<.001); SciSpace (β coefficient=–1.748; P<.001) and Elicit (β coefficient=–1.630; P<.001) had the lowest scores. Complex/scenario prompts produced significantly higher RHS than basic prompts (β coefficient=0.486; P<.001).
  • SciSpace and Elicit had the highest correct reference identifiers (629 and 597, respectively); ChatGPT had the highest hallucination results (592), and Bing had the highest rate of failure to generate results (210).

Methods

  • RHS was developed using a literature review and Delphi technique: 2 senior librarians, 12 senior physicians, and 11 junior physicians were consulted, followed by author consensus. Seven items were scored: erroneous title, journal name, authors’ names, or DOI scored 2; erroneous publication date, web link, or relevance to prompt keywords scored 1; maximum RHS=11, minimum=0.
  • Six AI chatbots (ChatGPT 3.5, Bard, Perplexity, Bing, Elicit, SciSpace) received the same 10 medical prompts (5 topics, each in basic and complex/scenario format), requesting 10 references per prompt with 7 bibliographic items. References were verified against PubMed first, then Google Scholar.
  • Analysis used mean/SD or median/IQR, Kruskal-Wallis and Mann-Whitney tests, and multivariable generalized linear gamma regression for mean RHS by chatbot, prompt complexity, and prompt iteration; SPSS was used with α=.05. No institutional review board approval was required.

Limitations

  • The authors state the RHS is novel and liable to future improvement; it includes a limited number of bibliographic parameters, and other researchers might consider additional identifiers crucial.
  • The authors state prompt structure is a limitation because LLM output depends heavily on prompt design, possibly explaining Bard’s failure, and the medical prompts may have limited scientific scope; verification via PubMed and Google Scholar may be suboptimal, only 6 chatbots were assessed, and selection biases may affect generalizability.
  • Evident from the text: the paper does not report inter-rater reliability or duplicate independent verification for the reference verification/RHS scoring process.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/54345

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

15 Valid · 0 Not valid · 1 Not checked · 97 Own results and interpretations

50 more claims

Create a free account to see all 113 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 32
  • Probably not a substantial claim. 37
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

3 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 39 references

Cited by

Items 101–176 of 176

Metadata

Authors
Aljamaan, F., Temsah, M.-H., Altamimi, I., Al-Eyadhy, A., Jamal, A., Alhasan, K., Mesallam, T. A., Farahat, M., & Malki, K. H.
Affiliations
King Saud University (SA)
Corresponding author
Malki, Khalid H
Venue
JMIR Medical Informatics
Published
2024-07-31
Volume
12
Pages
e54345
ISSN
2291-9694 (ISSN-L)
OpenAlex
W4400270729
Cited by
211
References
37
Citations by year
2026: 130 · 2025: 70 · 2024: 11
FWCI
17.16
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Medical Informatics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
AI chatbots, hallucinations, citation evaluation, bibliographic references, ChatGPT, Bing Chat, perplexity, Elicit, SciSpace, prompt complexity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.