The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index.
A Culturally Sensitive Test to Evaluate Nuanced GPT Hallucination
Open accessAbstract
The Generative Pre-trained Transformer(GPT) models, renowned for generating human-like text, occasionally produce “hallucinations” - outputs that diverge from human expectations. Current mitigation strategies for these GPT hallucinations largely rely on algorithmic automation, thereby overlooking the complexities of human judgment and cultural influence, particularly in fact interpretation. Addressing this issue, we have introduced a Culturally Sensitive Test that integrates language subjectivity, cultural nuances, and GPT idiosyncrasies. We have applied this test to five GPT models—OpenAI’s ChatGPT-3.5 and ChatGPT-4, Google’s Bard, Perplexity AI and TruthGPT - evaluating their responses to 70 questions across seven categories designed to provoke hallucinations. The evaluated models demonstrated varying performance, with controversial topics, those lacking clear scientific consensus and the brain teasers proving more susceptible to GPT hallucinations. Our study has paved the way for a nuanced assessment of GPT hallucinations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our review also found that only 6 of the 23 surveyed benchmark studies ( [7], [10], [31], [39], [44], [52]) were peerreviewed at the time of this article, reflecting the early-stage nature of research in this domain.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“PromptBench [7]: The study demonstrated that LLMs were not robust to adversarial prompts, highlighting a significant gap in evaluating models' performance in real-world, context-specific scenarios.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“PromptBench [7]: The study acknowledged the challenge of discerning genuine reasoning from technical optimization in LLMs, indicating that it primarily evaluated models based on their ability to follow instructions and generate outputs without deeply investigating whether these outputs resulted from true understanding or merely optimization strategies.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“PromptBench [7]: The study did not establish a clear criterion for balancing helpfulness and harmlessness in adversarial prompt evaluation, focusing instead on the robustness of LLMs against adversarial prompts without directly addressing how these models balance being helpful and avoiding harmful outputs in real-world scenarios.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study encountered difficulties in efficiently scaling their benchmark framework, Prompt-Bench, across various large language models, indicating substantial technical challenges in installation and scalability.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study did not address the potential variability in implementing its adversarial prompt attacks across different models or laboratories, and it did not attempt to standardize or ensure uniform implementation methodologies for these prompts.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study required iterating over the entire dataset 100 times on average to generate one adversarial prompt, indicating a potentially prolonged evaluation process, which aligned with the mentioned inadequacy of slow iteration time in benchmarks.The study did not specifically address or propose solutions to reduce these iteration time, suggesting that the challenge of aligning rapid AI development”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study focused on creating adversarial prompts to test LLM robustness but did not address the integration of diverse cultural and ideological norms, which could affect standardized answers' validity across different cultures and ideologies.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study did not address the diversity of human curators and evaluators, focusing instead on generating adversarial prompts across multiple levels to test LLMs' robustness, which did not directly relate to or account for the variability in human interpretation and judgment based on cultural, religious, political, and academic or commercial backgrounds.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
Citations
Cites
1 retracted
Zhang, J., Pu, J., Xue, J., Yang, M. Z., Xu, X., Wang, X., & Wang, F. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3256982 · Cited by 71
RetractedLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 2×
“Building upon Alan Turing’s seminal work “Computing Ma-chinery and Intelligence” in 1950 [51], we propose a modifica-tion to the original Turing Test for evaluating hallucinations in GPT models.”
MethodSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · cited 4×
“While existing mitigation measures such as censorship or ambiguous responses can compromise AI systems’ accuracy and utility [11], [15], [31], they highlight the need for a more comprehensive, ethically informed assessment.”
BackgroundInfluentialGPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 2,330
Notes towards a description of social representations.
Moscovici, S. (1988) · European Journal of Social Psychology · doi:10.1002/ejsp.2420180303 · Cited by 2,003
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24750 · Cited by 919
ChatGPT: Jack of all trades, master of none
Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101861 · Cited by 758 · cited 1×
“Despite their usefulness, existing classifications often fall short due to the subjectivity of language interpretation [28], [29].”
BackgroundSame Facts, Different Interpretations: Partisan Motivation and Opinion on Iraq
Gaines, B. J., Kuklinski, J. H., Quirk, P. J., Peyton, B., & Verkuilen, J. (2007) · The Journal of Politics · doi:10.1111/j.1468-2508.2007.00601.x · Cited by 683
An Overview of Artificial Intelligence Ethics
Huang, C., Zhang, Z., Mao, B., & Yao, X. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3194503 · Cited by 649
Using cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559 · cited 2×
“The reflective cognition principle (RCP) serves as a robust framework to parallel human and AI cognition, particularly with respect to the generation and interpretation of information [48], [49].”
BackgroundAthaluri, S. A., Manthena, S. V., Kesapragada, V. S. R. K. M., Yarlagadda, V., Dave, T., & Duddumpudi, R. T. S. (2023) · Cureus · doi:10.7759/cureus.37432 · Cited by 403
Faithful to the Original: Fact Aware Neural Abstractive Summarization
Cao, Z., Wei, F., Li, W., & Li, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11912 · Cited by 369
AI-Generated Medical Advice—GPT and Beyond
Haupt, C. E., & Marks, M. (2023) · JAMA · doi:10.1001/jama.2023.5321 · Cited by 346
The ‘Real-World Approach’ and Its Problems: A Critique of the Term Ecological Validity
Holleman, G. A., Hooge, I. T. C., Kemner, C., & Hessels, R. S. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.00721 · Cited by 345
YOU JUST BELIEVE THAT BECAUSE…1
White, R. M. (2010) · Philosophical Perspectives · doi:10.1111/j.1520-8583.2010.00204.x · Cited by 270
Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I., & Gurevych, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1213 · Cited by 265
Culture and Hallucinations: Overview and Future Directions
Larøi, F., Luhrmann, T. M., Bell, V., Christian, W. A., Deshpande, S. N., Fernyhough, C., JENKINS, J. H., & Woods, A. (2014) · Schizophrenia Bulletin · doi:10.1093/schbul/sbu012 · Cited by 252
WARP: Word-level Adversarial ReProgramming
Hambardzumyan, K., Khachatrian, H., & May, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.381 · Cited by 243
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238
What Does ChatGPT Say: The DAO from Algorithmic Intelligence to Linguistic Intelligence
Wang, F., Miao, Q., Li, X., Wang, X., & Lin, Y. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123486 · Cited by 214
Assessing The Factual Accuracy of Generated Text
Goodrich, B., Rao, V., Liu, P. J., & Saleh, M. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330955 · Cited by 162
Artificial intelligence hallucinations
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04473-y · Cited by 136
Factual Error Correction for Abstractive Summarization Models
Cao, M., Dong, Y., Wu, J., & Cheung, J. C. K. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.506 · Cited by 132
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Schuster, T., Fisch, A., & Barzilay, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.52 · Cited by 126
Cognitive reflection, cognitive intelligence, and cognitive abilities: A meta-analysis
Otero, I., Salgado, J. F., & Moscoso, S. (2021) · Intelligence · doi:10.1016/j.intell.2021.101614 · Cited by 126
Detecting Hallucinated Content in Conditional Neural Sequence Generation
Zhou, C., Neubig, G., Gu, J., Diab, M., Guzmán, F., Zettlemoyer, L., & Ghazvininejad, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.120 · Cited by 122
A Mathematical Investigation of Hallucination and Creativity in GPT Models
Lee, M. (2023) · Mathematics · doi:10.3390/math11102320 · Cited by 105
Reducing Quantity Hallucinations in Abstractive Summarization
Zhao, Z., Cohen, S. B., & Webber, B. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.203 · Cited by 99
Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding
Dziri, N., Madotto, A., Zaïane, O., & Bose, A. J. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.168 · Cited by 81
DAO to HANOI via DeSci: AI Paradigm Shifts from AlphaGo to ChatGPT
Miao, Q., Zheng, W., Lv, Y., Huang, M., Ding, W., & Wang, F.-Y. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123561 · Cited by 78
Masters, K. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2208731 · Cited by 76
Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference
Dušek, O., & Kasner, Z. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.19 · Cited by 66
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation
Liu, T., Zhang, Y., Brockett, C., Mao, Y., Sui, Z., Chen, W., & Dolan, B. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.464 · Cited by 63
Self-citationFocus Attention: Promoting Faithfulness and Diversity in Summarization
Aralikatte, R., Narayan, S., Maynez, J., Rothe, S., & McDonald, R. (2021) · doi:10.18653/v1/2021.acl-long.474 · Cited by 45
McIlroy-Young, R., Kleinberg, J. M., Sen, S., Barocas, S., & Anderson, A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534177 · Cited by 31
Identifying Fluently Inadequate Output in Neural and Statistical Machine Translation.
Martindale, M. J., Carpuat, M. J., Duh, K., & McNamee, P. (2019) · Cited by 22
Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View
Liu, T., Zheng, X., Chang, B., & Sui, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17583 · Cited by 21
Self-citation(2021) · The Research Repository @ WVU (West Virginia University) · Cited by 19
Heidenreich, H. S., & Williams, J. R. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462578 · Cited by 11
Ji, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
‘An Infinite Deal of Nothing’: critical ruminations on ChatGPT and the politics of language
Roy, N., & Maity, M. (2023) · DECISION · doi:10.1007/s40622-023-00342-3 · Cited by 6
Watters, P., & Patel, M. (1999) · PubMed · Cited by 5
Self-citationThapa, N., Abidi, A. I., & Sunil Kumar, G. (2022) · International Conference on Computing for Sustainable Global Development · doi:10.23919/indiacom54597.2022.9763225 · Cited by 3
Identifying Untrustworthy Samples
Shen, L., Zhan, H., Shen, X., Chen, H., Zhao, X., & Zhu, X. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482352 · Cited by 2
Research works printed but not linked: 3
- Clayton, 2023, Sam Altman: CEO of OpenAI calls for us to regulate artificial intelligence
- Smith, 2023, Hallucinations could blunt ChatGPTs success
- Nichols, 2023, Machines hacking machines—Turings legacy [Carter], Cyber-Human Systems, Space Technologies, and Threats
Cited by
Sorted by citations
Game changers: A generative AI prompt protocol to enhance human-AI knowledge co-construction
Robertson, J., Ferreira, C. C., Botha, E., & Oosthuizen, K. (2024) · Business Horizons · doi:10.1016/j.bushor.2024.04.008 · Cited by 161
Generative AI Prompt Engineering for Educators: Practical Strategies
Park, J., & Choo, S. (2024) · Journal of Special Education Technology · doi:10.1177/01626434241298954 · Cited by 159
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102
“This technology has not only revolutionized the industry and academia, but has also reignited critical discussions concerning AI consciousness and its potential threats to humanity [9], [10], [11].”
BackgroundSelf-citationA Systematic Review of the Limitations and Associated Opportunities of ChatGPT
Cong‐Lem, N., Soyoof, A., & Tsering, D. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2344142 · Cited by 140
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104
“This lack of consideration results in benchmarks that wrongfully assume a uniform cognitive framework, failing to capture how LLMs handle linguistic variations across different cultures and languages [13].”
BackgroundInfluentialSelf-citation
Metadata
- Authors
- McIntosh, T. R., Liu, T., Susnjak, T., Watters, P., Ng, A., & Halgamuge, M. N.
- Affiliations
- La Trobe University (AU)
- Massey University (NZ)
- Massey University, Auckland, New Zealand
- Cyberstronomy Pty Ltd, Ballarat, VIC, Australia
- RMIT University (AU)
- Venue
- IEEE Transactions on Artificial Intelligence
- Published
- 2024-06
- Volume
- 5
- Issue
- 6
- Pages
- 2739-2751
- ISSN
- 2691-4581 (ISSN-L)
- OpenAlex
- W4388691793
- Cited by
- 145
- References
- 49
- Citations by year
- 2026: 12 · 2025: 31 · 2024: 90
- FWCI
- 13.67
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Licence
- Locations
- IEEE Transactions on Artificial Intelligence (publishedVersion)
- Figshare (submittedVersion, cc-by)
- La Trobe University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- hallucination evaluation, ChatGPT-4, Google Bard, Perplexity AI, controversial issues, brain teasers, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
