Geographic Variation in LLM DOI Fabrication: Cross-Country Analysis of Citation Accuracy Across Four Large Language Models

TL;DR

This study evaluates DOI hallucination in citations generated by four LLMs across ten countries and finds systematic geographic disparities, with higher fabrication rates in lower-income contexts and for recent publications. Model choice showed the strongest association with hallucination, followed by country income level and publication period.

Key findings

• Across 3451 generated citations validated via CrossRef, DOI hallucination increased as income level decreased; rates exceeded 80% in lower-income countries, and Gemini-2.0-flash-lite approached 100% hallucination for Bangladesh.
• Model-specific factors had the strongest association with DOI hallucination (Cramér’s V = 0.203), followed by country income level (V = 0.162) and geographic bias (V = 0.189); publication period was also significant (Kruskal-Wallis H = 161.74).
• Hallucination rates rose sharply for publications from the 2020s: high- and upper-middle-income countries increased from roughly 45–55% in earlier periods to 65–75%, while lower-middle-income countries rose from around 70% to 80%.
• Citation completeness clustered between 4.85–5.0, but India and Bangladesh showed both near-5.0 completeness and hallucination rates exceeding 80%, while the UK and South Korea had high completeness with lower hallucination rates around 48% and 50%.

Methods

• Controlled experimental design using GPT-4o-mini, Claude-3-haiku, Gemini-2.0-flash-lite, and DeepSeek V3, all configured with temperature 0.2 and a 2000-token limit; ten countries were purposively sampled by World Bank FY2024 income classification.
• Standardized system prompt plus five user prompts targeting information behavior categories (seeking, use, sharing, needs, general behavior), each requesting 20 real citations with valid DOIs in tab-delimited format; 195 experimental conditions were examined (10 countries × 5 prompts × 4 models, with OpenAI having 45 conditions due to technical constraints).
• DOIs were validated with the CrossRef REST API; HTTP 200 with complete bibliographic metadata was classified as valid, while error responses or missing metadata were classified as hallucinated. Analysis used Fisher’s exact tests with Monte Carlo simulation (2000 iterations), Kruskal-Wallis tests, Spearman correlations, and effect sizes including Cramér’s V.

Limitations

• The paper states that causal mechanisms cannot be established due to proprietary training data, restricting conclusions to systematic associations.
• The paper states that DOI validation focused only on resolution success and geographic relevance, not whether valid DOIs matched cited content, likely underestimating citation fabrication; it also notes exclusion of low-income countries, inclusion of only three lower-middle-income countries versus seven upper-middle- and high-income countries, fixed model versions and time period, restriction to information behavior research in LIS, and operationalization of hallucination solely as DOI resolution failure.
• Evident from the text: the study’s own sampling bias toward better-resourced contexts and narrow DOI-only validation mean the reported geographic disparities are likely conservative, as the authors also state.