Hallucination Rate of Peer-Reviewed Citations Generated by Large Language Models in Neurocritical Care

TL;DR

In a cross-sectional blinded evaluation of 300 references generated by GPT-5.3, DeepSeek-V3, and Grok-4 across 10 neurocritical care topics under zero-shot, retrieval-disabled web-interface conditions, 55.0% contained citation inaccuracies and 28.3% were completely fabricated. DeepSeek-V3 had the lowest hallucination and fabrication rates, but all tested models produced citations unreliable without cross-database verification.

Key findings

• Overall, 165 of 300 references (55.0%) contained a citation inaccuracy, and 85 of 300 (28.3%) were completely fabricated; inter-rater agreement was excellent (κ = 0.91; 95% CI, 0.86–0.96).
• Model-level rates: DeepSeek-V3 had 23% hallucination and 8% fabrication; GPT-5.3 had 69% hallucination and 27% fabrication; Grok-4 had 73% hallucination and 50% fabrication. High-severity inaccuracies (score ≥ 2) occurred in 10% of DeepSeek outputs, 44% of GPT outputs, and 66% of Grok outputs.
• Compared with DeepSeek-V3, GPT-5.3 was 3.00 times more likely to hallucinate (95% CI, 1.94–4.63; p < 0.001) and Grok-4 was 3.17 times more likely (95% CI, 2.03–4.96; p < 0.001). For fabrication, GPT had RR 3.38 (95% CI, 1.52–7.49; p = 0.003) and Grok had RR 6.25 (95% CI, 3.07–12.71; p < 0.001) versus DeepSeek; Grok versus GPT for fabrication was not statistically significant (RR 1.85; 95% CI, 0.95–3.59; p = 0.069).
• Identifier omission increased with citation severity: 14.1% of accurate references had no identifier versus 54.1% of fabricated references (χ² = 79.07, p < 0.001); 3.5% of fabricated references carried a correctly resolving identifier.

Methods

• Cross-sectional, blinded technology performance evaluation: 10 neurocritical care topics were submitted to GPT-5.3, DeepSeek-V3, and Grok-4 on March 10, 2026, using identical zero-shot prompts with retrieval/browsing disabled via official paid web interfaces; each model generated 10 recent (2020 or later) peer-reviewed references per topic, yielding 300 references.
• Two blinded neurocritical care experts independently verified each reference against PubMed, DOI resolver, Google Scholar, and CrossRef and scored it using a modified Reference Hallucination Scale (0 = fully accurate, 1 = minor error, 2 = major error, 3 = completely fabricated); discrepancies were resolved by consensus. Primary outcome was any hallucination (score ≥ 1); secondary outcome was fabrication (score = 3).
• Analyses included Cohen kappa for inter-rater reliability, Kruskal-Wallis with Dunn post hoc testing, modified Poisson regression with robust SEs clustered at the model-topic level for relative risks, and Fisher exact tests for topic-level comparisons; analyses used Python 3.12.

Limitations

• The authors state that the study used a single zero-shot prompt without chain-of-thought or retrieval-augmented generation, tested models at a single time point, selected only 10 topics by one domain expert without formal consensus or Delphi process, did not stratify topics by evidence density or guideline availability, could not exactly replicate stochastic outputs, used commercial web interfaces that limit control over temperature/system prompts/backend routing, primarily assessed bibliographic accuracy rather than topical relevance or clinical usefulness, sampled each model-topic combination once, and had underpowered topic-level analyses.
• Evident from the design: no repeated sampling per model-topic cell, no retrieval-enabled comparator arm, and no separate relevance or evidentiary-quality scoring were included, so the reported rates reflect one constrained zero-shot snapshot rather than optimized or repeated deployment conditions.