The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index.
A Culturally Sensitive Test to Evaluate Nuanced GPT Hallucination
Open accessAbstract
The Generative Pre-trained Transformer(GPT) models, renowned for generating human-like text, occasionally produce “hallucinations” - outputs that diverge from human expectations. Current mitigation strategies for these GPT hallucinations largely rely on algorithmic automation, thereby overlooking the complexities of human judgment and cultural influence, particularly in fact interpretation. Addressing this issue, we have introduced a Culturally Sensitive Test that integrates language subjectivity, cultural nuances, and GPT idiosyncrasies. We have applied this test to five GPT models—OpenAI’s ChatGPT-3.5 and ChatGPT-4, Google’s Bard, Perplexity AI and TruthGPT - evaluating their responses to 70 questions across seven categories designed to provoke hallucinations. The evaluated models demonstrated varying performance, with controversial topics, those lacking clear scientific consensus and the brain teasers proving more susceptible to GPT hallucinations. Our study has paved the way for a nuanced assessment of GPT hallucinations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our review also found that only 6 of the 23 surveyed benchmark studies ( [7], [10], [31], [39], [44], [52]) were peerreviewed at the time of this article, reflecting the early-stage nature of research in this domain.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“PromptBench [7]: The study demonstrated that LLMs were not robust to adversarial prompts, highlighting a significant gap in evaluating models' performance in real-world, context-specific scenarios.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“PromptBench [7]: The study acknowledged the challenge of discerning genuine reasoning from technical optimization in LLMs, indicating that it primarily evaluated models based on their ability to follow instructions and generate outputs without deeply investigating whether these outputs resulted from true understanding or merely optimization strategies.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“PromptBench [7]: The study did not establish a clear criterion for balancing helpfulness and harmlessness in adversarial prompt evaluation, focusing instead on the robustness of LLMs against adversarial prompts without directly addressing how these models balance being helpful and avoiding harmful outputs in real-world scenarios.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study encountered difficulties in efficiently scaling their benchmark framework, Prompt-Bench, across various large language models, indicating substantial technical challenges in installation and scalability.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study did not address the potential variability in implementing its adversarial prompt attacks across different models or laboratories, and it did not attempt to standardize or ensure uniform implementation methodologies for these prompts.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study required iterating over the entire dataset 100 times on average to generate one adversarial prompt, indicating a potentially prolonged evaluation process, which aligned with the mentioned inadequacy of slow iteration time in benchmarks.The study did not specifically address or propose solutions to reduce these iteration time, suggesting that the challenge of aligning rapid AI development”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study focused on creating adversarial prompts to test LLM robustness but did not address the integration of diverse cultural and ideological norms, which could affect standardized answers' validity across different cultures and ideologies.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study did not address the diversity of human curators and evaluators, focusing instead on generating adversarial prompts across multiple levels to test LLMs' robustness, which did not directly relate to or account for the variability in human interpretation and judgment based on cultural, religious, political, and academic or commercial backgrounds.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
Citations
Cites
1 retracted
Zhang, J., Pu, J., Xue, J., Yang, M. Z., Xu, X., Wang, X., & Wang, F. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3256982 · Cited by 71
RetractedLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 2×
“Building upon Alan Turing’s seminal work “Computing Ma-chinery and Intelligence” in 1950 [51], we propose a modifica-tion to the original Turing Test for evaluating hallucinations in GPT models.”
MethodSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · cited 4×
“While existing mitigation measures such as censorship or ambiguous responses can compromise AI systems’ accuracy and utility [11], [15], [31], they highlight the need for a more comprehensive, ethically informed assessment.”
BackgroundInfluentialGPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 2,330
Notes towards a description of social representations.
Moscovici, S. (1988) · European Journal of Social Psychology · doi:10.1002/ejsp.2420180303 · Cited by 2,003
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24750 · Cited by 919
Research works printed but not linked: 3
- Clayton, 2023, Sam Altman: CEO of OpenAI calls for us to regulate artificial intelligence
- Smith, 2023, Hallucinations could blunt ChatGPTs success
- Nichols, 2023, Machines hacking machines—Turings legacy [Carter], Cyber-Human Systems, Space Technologies, and Threats
Cited by
Sorted by citations
Game changers: A generative AI prompt protocol to enhance human-AI knowledge co-construction
Robertson, J., Ferreira, C. C., Botha, E., & Oosthuizen, K. (2024) · Business Horizons · doi:10.1016/j.bushor.2024.04.008 · Cited by 161
Generative AI Prompt Engineering for Educators: Practical Strategies
Park, J., & Choo, S. (2024) · Journal of Special Education Technology · doi:10.1177/01626434241298954 · Cited by 159
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102
“This technology has not only revolutionized the industry and academia, but has also reignited critical discussions concerning AI consciousness and its potential threats to humanity [9], [10], [11].”
BackgroundSelf-citationA Systematic Review of the Limitations and Associated Opportunities of ChatGPT
Cong‐Lem, N., Soyoof, A., & Tsering, D. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2344142 · Cited by 140
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104
“This lack of consideration results in benchmarks that wrongfully assume a uniform cognitive framework, failing to capture how LLMs handle linguistic variations across different cultures and languages [13].”
BackgroundInfluentialSelf-citation
Metadata
- Authors
- McIntosh, T. R., Liu, T., Susnjak, T., Watters, P., Ng, A., & Halgamuge, M. N.
- Affiliations
- La Trobe University (AU)
- Massey University (NZ)
- Massey University, Auckland, New Zealand
- Cyberstronomy Pty Ltd, Ballarat, VIC, Australia
- RMIT University (AU)
- Venue
- IEEE Transactions on Artificial Intelligence
- Published
- 2024-06
- Volume
- 5
- Issue
- 6
- Pages
- 2739-2751
- ISSN
- 2691-4581 (ISSN-L)
- OpenAlex
- W4388691793
- Cited by
- 145
- References
- 49
- Citations by year
- 2026: 12 · 2025: 31 · 2024: 90
- FWCI
- 13.67
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Licence
- Locations
- IEEE Transactions on Artificial Intelligence (publishedVersion)
- Figshare (submittedVersion, cc-by)
- La Trobe University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- hallucination evaluation, ChatGPT-4, Google Bard, Perplexity AI, controversial issues, brain teasers, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
