The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index.
A Culturally Sensitive Test to Evaluate Nuanced GPT Hallucination
Open accessAbstract
The Generative Pre-trained Transformer(GPT) models, renowned for generating human-like text, occasionally produce “hallucinations” - outputs that diverge from human expectations. Current mitigation strategies for these GPT hallucinations largely rely on algorithmic automation, thereby overlooking the complexities of human judgment and cultural influence, particularly in fact interpretation. Addressing this issue, we have introduced a Culturally Sensitive Test that integrates language subjectivity, cultural nuances, and GPT idiosyncrasies. We have applied this test to five GPT models—OpenAI’s ChatGPT-3.5 and ChatGPT-4, Google’s Bard, Perplexity AI and TruthGPT - evaluating their responses to 70 questions across seven categories designed to provoke hallucinations. The evaluated models demonstrated varying performance, with controversial topics, those lacking clear scientific consensus and the brain teasers proving more susceptible to GPT hallucinations. Our study has paved the way for a nuanced assessment of GPT hallucinations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our review also found that only 6 of the 23 surveyed benchmark studies ( [7], [10], [31], [39], [44], [52]) were peerreviewed at the time of this article, reflecting the early-stage nature of research in this domain.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“PromptBench [7]: The study demonstrated that LLMs were not robust to adversarial prompts, highlighting a significant gap in evaluating models' performance in real-world, context-specific scenarios.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“PromptBench [7]: The study acknowledged the challenge of discerning genuine reasoning from technical optimization in LLMs, indicating that it primarily evaluated models based on their ability to follow instructions and generate outputs without deeply investigating whether these outputs resulted from true understanding or merely optimization strategies.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“PromptBench [7]: The study did not establish a clear criterion for balancing helpfulness and harmlessness in adversarial prompt evaluation, focusing instead on the robustness of LLMs against adversarial prompts without directly addressing how these models balance being helpful and avoiding harmful outputs in real-world scenarios.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study encountered difficulties in efficiently scaling their benchmark framework, Prompt-Bench, across various large language models, indicating substantial technical challenges in installation and scalability.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study did not address the potential variability in implementing its adversarial prompt attacks across different models or laboratories, and it did not attempt to standardize or ensure uniform implementation methodologies for these prompts.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study required iterating over the entire dataset 100 times on average to generate one adversarial prompt, indicating a potentially prolonged evaluation process, which aligned with the mentioned inadequacy of slow iteration time in benchmarks.The study did not specifically address or propose solutions to reduce these iteration time, suggesting that the challenge of aligning rapid AI development”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study focused on creating adversarial prompts to test LLM robustness but did not address the integration of diverse cultural and ideological norms, which could affect standardized answers' validity across different cultures and ideologies.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study did not address the diversity of human curators and evaluators, focusing instead on generating adversarial prompts across multiple levels to test LLMs' robustness, which did not directly relate to or account for the variability in human interpretation and judgment based on cultural, religious, political, and academic or commercial backgrounds.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
Citations
Cites
1 retracted
Zhang, J., Pu, J., Xue, J., Yang, M. Z., Xu, X., Wang, X., & Wang, F. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3256982 · Cited by 71
RetractedLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 2×
“Building upon Alan Turing’s seminal work “Computing Ma-chinery and Intelligence” in 1950 [51], we propose a modifica-tion to the original Turing Test for evaluating hallucinations in GPT models.”
MethodSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · cited 4×
“While existing mitigation measures such as censorship or ambiguous responses can compromise AI systems’ accuracy and utility [11], [15], [31], they highlight the need for a more comprehensive, ethically informed assessment.”
BackgroundInfluentialGPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 2,330
Notes towards a description of social representations.
Moscovici, S. (1988) · European Journal of Social Psychology · doi:10.1002/ejsp.2420180303 · Cited by 2,003
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24750 · Cited by 919
Research works printed but not linked: 3
- Clayton, 2023, Sam Altman: CEO of OpenAI calls for us to regulate artificial intelligence
- Smith, 2023, Hallucinations could blunt ChatGPTs success
- Nichols, 2023, Machines hacking machines—Turings legacy [Carter], Cyber-Human Systems, Space Technologies, and Threats
Cited by
Sorted by citations
Items 101–146 of 146
Assessing Audio Hallucination in Large Multimodal ModelsPreprint
Hanamaki, S., Kirishima, N., & Narumi, S. (2024) · doi:10.31219/osf.io/f8ra7 · Cited by 4
Tremblay, M., Gervais, S., & Maisonneuve, D. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/bzq8h · Cited by 4
Lainwright, N., & Pemberton, M. (2024) · doi:10.31219/osf.io/34yqj · Cited by 4
Automated Learning of Fine-Grained Citation Patterns in Open Source Large Language ModelsPreprint
Harcourt, E., Loxley, J., & Stanson, B. (2024) · doi:10.31219/osf.io/xnesr · Cited by 4
Analysis of Bias in GPT Language Models through Fine-tuning Containing Divergent Data
Turi, L. F., Cavalini, A., Comarela, G., Oliveira-Santos, T., Badue, C. S., & Souza, A. F. D. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650574 · Cited by 3
Educational Technology and Responsible Automated Essay Scoring in the Generative AI Era
Trung, H. T., Nguyen, M. N., Nguyen, T. T. H., Vo, D. T. H., Thanh, B. N., Nguyen, K., Ha, S., & Le, T. V. A. (2024) · Practice, progress, and proficiency in sustainability · doi:10.4018/979-8-3693-2827-9.ch011 · Cited by 3
Fazelnia, M., Mirakhorli, M. T., & Bagheri, H. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695302 · Cited by 3
Mitigating Structural Hallucination in Large Language Models with Local DiffusionPreprint
Kiritani, K., & Kayano, T. (2024) · Research Square · doi:10.21203/rs.3.rs-4678127/v1 · Cited by 3
Guest Editorial Education in the World of ChatGPT and Generative AI
Tan, S. C., Wijekumar, K., Hong, H., Olmanson, J., Twomey, R., & Sinha, T. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3451050 · Cited by 3
Efficient Conceptual Knowledge Removal in Large Language Models: Methods and EvaluationsPreprint
Dimitriou, M., Rogowski, D., Anderson, M., Vanderbilt, E., & Carmichael, L. (2024) · Research Square · doi:10.21203/rs.3.rs-5208091/v1 · Cited by 3
Mapping Generative Artificial Intelligence (GAI's) Exciting Future: From Gemini to Q* and Beyond
Akhtar, Z. B. (2024) · EAI Endorsed Transactions on AI and Robotics · doi:10.4108/airo.5962 · Cited by 4
Bucchiarone, A., Bonetti, F., & Yigitbas, E. (2025) · International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) · doi:10.1109/seams66627.2025.00019 · Cited by 2
Liu, T., McIntosh, T. R., Sušnjak, T., Watters, P., & Halgamuge, M. N. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3757927 · Cited by 2
Self-citationMadhee, D. J. M. A., Zaid, M. I., Atshan, N. A., Ali, K. A. M., & Jaboob, A. S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-15806-2_31 · Cited by 2
Can Generative AI Support Educators? Creating Learning Paths with PolyGloT
Bucchiarone, A., Gini, F., Bonetti, F., Bassanelli, S., Schiavo, G., Martorella, T., Adami, F., Guidolin, T., & Zambotto, L. (2024) · Springer eBooks · doi:10.1007/978-3-031-65691-0_20 · Cited by 2
Trust or distrust? AIGC trustworthiness and an extended analysis within nomology framework
Li, M., Du, R., Ai, S., Hunt, R. A., & Xie, J. (2025) · Data Science and Management · doi:10.1016/j.dsm.2025.09.005 · Cited by 3
Retrieval Augmented Generation via Context Compression Techniques for Large Language ModelsPreprint
Jiang, P., Fan, R., & Yong, Y. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ua6j5 · Cited by 3
The Proficiency Paradox: Generative AI, Governance Capacity, and Knowledge-Sharing SystemsPreprint
Shankar, R., & Sim, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4867815 · Cited by 2
Gupta, S., Agrawal, V., Negi, Y., Karunanithi, S., & Balakrishnan, A. (2026) · IEEE Access · doi:10.1109/access.2026.3675624 · Cited by 2
Bane and Boon of Hallucinations in the Context of Generative AI
Sakib, S. M. N. (2024) · Advances in business information systems and analytics book series · doi:10.4018/979-8-3693-2643-5.ch016 · Cited by 4
Co-creation with Generative Artificial Intelligence in the Development of Digital Interfaces
Correia, A. R., & Pereira, J. B. (2025) · Smart innovation, systems and technologies · doi:10.1007/978-981-96-3077-6_15 · Cited by 1
Socioeconomic Threats of Deepfakes and the Role of Cyber-Wellness Education in Defense
Ahvanooey, M. T., Mazurczyk, W., & Lee, D. (2025) · Communications of the ACM · doi:10.1145/3715317 · Cited by 2
Halucynacje chatbotów a prawda: główne nurty debaty i ich interpretacje
Kreft, J., Boguszewicz-Kreft, M., & Cyrek, B. (2024) · Roczniki Nauk Społecznych · doi:10.18290/rns2024.0009 · Cited by 2
Culturally Sensitive Health Solutions: Engineering Localized Practices
Bizimana Rukundo, T. (2025) · IAA Journal of Biological Sciences · doi:10.59298/iaajb/2025/131102108 · Cited by 1
RansomCillin: Leveraging NTFS Spare Space to Recover from Ransomware AttacksPreprint
Takeuchi, K., Fujima, H., Kumamoto, T., & Yoshida, Y. (2023) · doi:10.36227/techrxiv.24596754 · Cited by 1
Semi-Structured Data Parsing and Vectorization Using Retrieval-Augmented GenerationPreprint
Cai, J., & Li, Y. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/qfkbv
Understanding How ChatGPT Memorizes Chinese Tang Poetry
Chang, C.-Y. (2024) · IEEE International Conference on Big Data Security on Cloud, IEEE International Conference on High Performance and Smart Computing, and IEEE International Conference on Intelligent Data and Security (BigDataSecurity/HPSC/IDS) · doi:10.1109/ids62739.2024.00008
Studying the Knowledge and Behavior of ChatGPT via Journey to the West
Zhou, Z. (2024) · IEEE International Conference on Big Data Security on Cloud, IEEE International Conference on High Performance and Smart Computing, and IEEE International Conference on Intelligent Data and Security (BigDataSecurity/HPSC/IDS) · doi:10.1109/hpsc62738.2024.00012
Ito, M., Nishikawa, H., & Sakamoto, Y. (2024) · Research Square · doi:10.21203/rs.3.rs-4814991/v1
Abdullah, M. H. A., Aziz, N., Abdulkadir, S. J., Hussain, K., Alhussian, H., & Talpur, N. (2024) · Journal of Data and Information Science · doi:10.2478/jdis-2024-0029
Adaptive Neural Contextualization for Expansive Knowledge RepresentationPreprint
Canus, S., Torrington, W., Northfield, M., & Weatherstone, C. (2024) · doi:10.31219/osf.io/uaev5
Sang, N. Q., Thao, N., Tam, D. T. M., & Duc, M. L. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5264269
Topham, L., Khan, W., Atherton, P., Khan, I., & Reynolds, T. (2025) · International Conference on Developments in eSystems Engineering (DeSE) · doi:10.1109/dese68208.2025.11367945
Younsoo, P., Park, H.-H., & Yi, Y. (2026) · The Journal of Korean Association of Computer Education · doi:10.32431/kace.2026.29.1.002
AI Hallucinations as an Existential Risk to Human Society: Mitigating the Risks
Goltz, N., & Yuxin, Z. (2026) · BENTHAM SCIENCE PUBLISHERS eBooks · doi:10.2174/9798898812256126040007
Hallucination Detection and Reclassification Using Graph Co-Training
Kihlman, R., & Fasli, M. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402538
Shamseera, M. K., & Durga, R. (2026) · International Conference on AI-Driven Smart Systems and Ubiquitous Computing (ICAUC) · doi:10.1109/icauc68182.2026.11441312
Introduction: Advancing Technical Editing in the Age of GAI
Jablonski, J., & Nagelhout, E. (2026) · Technical Communication Quarterly · doi:10.1080/10572252.2026.2651131
Enhancing Business Dashboards with Explanatory Analytics
Caron, E., & Trần, T. P. D. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-16234-2_2
Iarovoi, D., & Teh, P. L. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-34792-3_11
Dakuru, M., Lam, S. S., & Elluri, L. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-24360-7_26
Ahn, H., Oh, S., Kim, G. S., Park, S., Jung, S., & Kim, J. (2026) · IEEE Transactions on Networking · doi:10.1109/ton.2026.3725648
Conceptualising student-GenAI knowledge co-creation in higher education
Bui, H. T. (2026) · Innovations in Education and Teaching International · doi:10.1080/14703297.2026.2729359
Jafarian, M. (2026) · Behavior Research Methods · doi:10.3758/s13428-026-03207-6
Metadata
- Authors
- McIntosh, T. R., Liu, T., Susnjak, T., Watters, P., Ng, A., & Halgamuge, M. N.
- Affiliations
- La Trobe University (AU)
- Massey University (NZ)
- Massey University, Auckland, New Zealand
- Cyberstronomy Pty Ltd, Ballarat, VIC, Australia
- RMIT University (AU)
- Venue
- IEEE Transactions on Artificial Intelligence
- Published
- 2024-06
- Volume
- 5
- Issue
- 6
- Pages
- 2739-2751
- ISSN
- 2691-4581 (ISSN-L)
- OpenAlex
- W4388691793
- Cited by
- 145
- References
- 49
- Citations by year
- 2026: 12 · 2025: 31 · 2024: 90
- FWCI
- 13.67
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Licence
- Locations
- IEEE Transactions on Artificial Intelligence (publishedVersion)
- Figshare (submittedVersion, cc-by)
- La Trobe University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- hallucination evaluation, ChatGPT-4, Google Bard, Perplexity AI, controversial issues, brain teasers, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
