A Culturally Sensitive Test to Evaluate Nuanced GPT Hallucination
Open accessAbstract
The Generative Pre-trained Transformer(GPT) models, renowned for generating human-like text, occasionally produce “hallucinations” - outputs that diverge from human expectations. Current mitigation strategies for these GPT hallucinations largely rely on algorithmic automation, thereby overlooking the complexities of human judgment and cultural influence, particularly in fact interpretation. Addressing this issue, we have introduced a Culturally Sensitive Test that integrates language subjectivity, cultural nuances, and GPT idiosyncrasies. We have applied this test to five GPT models—OpenAI’s ChatGPT-3.5 and ChatGPT-4, Google’s Bard, Perplexity AI and TruthGPT - evaluating their responses to 70 questions across seven categories designed to provoke hallucinations. The evaluated models demonstrated varying performance, with controversial topics, those lacking clear scientific consensus and the brain teasers proving more susceptible to GPT hallucinations. Our study has paved the way for a nuanced assessment of GPT hallucinations.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Only 6 of the 23 surveyed benchmark studies were peer-reviewed at the time of this article.
“Our review also found that only 6 of the 23 surveyed benchmark studies ( [7], [10], [31], [39], [44], [52]) were peerreviewed at the time of this article, reflecting the early-stage nature of research in this domain.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
The PromptBench benchmark demonstrated that LLMs were not robust to adversarial prompts, highlighting a significant gap in evaluating models' performance in real-world, context-specific scenarios.
“PromptBench [7]: The study demonstrated that LLMs were not robust to adversarial prompts, highlighting a significant gap in evaluating models' performance in real-world, context-specific scenarios.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
The PromptBench benchmark acknowledged the challenge of discerning genuine reasoning from technical optimization in LLMs, indicating that it primarily evaluated models based on their ability to follow instructions and generate outputs without deeply investigating whether these outputs resulted from true understanding or merely optimization strategies.
“PromptBench [7]: The study acknowledged the challenge of discerning genuine reasoning from technical optimization in LLMs, indicating that it primarily evaluated models based on their ability to follow instructions and generate outputs without deeply investigating whether these outputs resulted from true understanding or merely optimization strategies.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
The PromptBench benchmark did not establish a clear criterion for balancing helpfulness and harmlessness in adversarial prompt evaluation, focusing instead on robustness of LLMs against adversarial prompts without directly addressing how these models balance being helpful and avoiding harmful outputs in real-world scenarios.
“PromptBench [7]: The study did not establish a clear criterion for balancing helpfulness and harmlessness in adversarial prompt evaluation, focusing instead on the robustness of LLMs against adversarial prompts without directly addressing how these models balance being helpful and avoiding harmful outputs in real-world scenarios.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
PromptBench encountered difficulties in efficiently scaling their benchmark framework across various large language models, indicating substantial technical challenges in installation and scalability.
“The study encountered difficulties in efficiently scaling their benchmark framework, Prompt-Bench, across various large language models, indicating substantial technical challenges in installation and scalability.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
PromptBench did not address the potential variability in implementing its adversarial prompt attacks across different models or laboratories, and did not attempt to standardize or ensure uniform implementation methodologies.
“The study did not address the potential variability in implementing its adversarial prompt attacks across different models or laboratories, and it did not attempt to standardize or ensure uniform implementation methodologies for these prompts.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
PromptBench required iterating over the entire dataset 100 times on average to generate one adversarial prompt, indicating a potentially prolonged evaluation process; it did not specifically address or propose solutions to reduce iteration time.
“The study required iterating over the entire dataset 100 times on average to generate one adversarial prompt, indicating a potentially prolonged evaluation process, which aligned with the mentioned inadequacy of slow iteration time in benchmarks.The study did not specifically address or propose solutions to reduce these iteration time, suggesting that the challenge of aligning rapid AI development”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
PromptBench focused on creating adversarial prompts to test LLM robustness but did not address integration of diverse cultural and ideological norms, which could affect standardized answers' validity across different cultures and ideologies.
“The study focused on creating adversarial prompts to test LLM robustness but did not address the integration of diverse cultural and ideological norms, which could affect standardized answers' validity across different cultures and ideologies.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
PromptBench did not address diversity of human curators and evaluators, focusing instead on generating adversarial prompts to test LLM robustness.
“The study did not address the diversity of human curators and evaluators, focusing instead on generating adversarial prompts across multiple levels to test LLMs' robustness, which did not directly relate to or account for the variability in human interpretation and judgment based on cultural, religious, political, and academic or commercial backgrounds.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
Cites
1 retracted
Zhang, J., Pu, J., Xue, J., Yang, M. Z., Xu, X., Wang, X., & Wang, F. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3256982 · Cited by 71 · see citing works
RetractedLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 2× · see citing works
“Building upon Alan Turing’s seminal work “Computing Ma-chinery and Intelligence” in 1950 [51], we propose a modifica-tion to the original Turing Test for evaluating hallucinations in GPT models.”
MethodSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · cited 4× · see citing works
“While existing mitigation measures such as censorship or ambiguous responses can compromise AI systems’ accuracy and utility [11], [15], [31], they highlight the need for a more comprehensive, ethically informed assessment.”
BackgroundInfluentialNotes towards a description of social representations.
Moscovici, S. (1988) · European Journal of Social Psychology · doi:10.1002/ejsp.2420180303 · Cited by 2,003 · see citing works
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · see citing works
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185 · see citing works
Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24750 · Cited by 919 · see citing works
ChatGPT: Jack of all trades, master of none
Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101861 · Cited by 758 · cited 1× · see citing works
“Despite their usefulness, existing classifications often fall short due to the subjectivity of language interpretation [28], [29].”
BackgroundSame Facts, Different Interpretations: Partisan Motivation and Opinion on Iraq
Gaines, B. J., Kuklinski, J. H., Quirk, P. J., Peyton, B., & Verkuilen, J. (2007) · The Journal of Politics · doi:10.1111/j.1468-2508.2007.00601.x · Cited by 683 · see citing works
An Overview of Artificial Intelligence Ethics
Huang, C., Zhang, Z., Mao, B., & Yao, X. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3194503 · Cited by 583 · see citing works
Using cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559 · cited 2× · see citing works
“The reflective cognition principle (RCP) serves as a robust framework to parallel human and AI cognition, particularly with respect to the generation and interpretation of information [48], [49].”
BackgroundGPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 545 · see citing works
Athaluri, S. A., Manthena, S. V., Kesapragada, V. S. R. K. M., Yarlagadda, V., Dave, T., & Duddumpudi, R. T. S. (2023) · Cureus · doi:10.7759/cureus.37432 · Cited by 403 · see citing works
Faithful to the Original: Fact Aware Neural Abstractive Summarization
Cao, Z., Wei, F., Li, W., & Li, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11912 · Cited by 359 · see citing works
AI-Generated Medical Advice—GPT and Beyond
Haupt, C. E., & Marks, M. (2023) · JAMA · doi:10.1001/jama.2023.5321 · Cited by 346 · see citing works
The ‘Real-World Approach’ and Its Problems: A Critique of the Term Ecological Validity
Holleman, G. A., Hooge, I. T. C., Kemner, C., & Hessels, R. S. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.00721 · Cited by 345 · see citing works
YOU JUST BELIEVE THAT BECAUSE…1
White, R. M. (2010) · Philosophical Perspectives · doi:10.1111/j.1520-8583.2010.00204.x · Cited by 270 · see citing works
Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I., & Gurevych, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1213 · Cited by 265 · see citing works
Culture and Hallucinations: Overview and Future Directions
Larøi, F., Luhrmann, T. M., Bell, V., Christian, W. A., Deshpande, S. N., Fernyhough, C., JENKINS, J. H., & Woods, A. (2014) · Schizophrenia Bulletin · doi:10.1093/schbul/sbu012 · Cited by 252 · see citing works
WARP: Word-level Adversarial ReProgramming
Hambardzumyan, K., Khachatrian, H., & May, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.381 · Cited by 243 · see citing works
<scp>SummaC</scp>: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238 · see citing works
What Does ChatGPT Say: The DAO from Algorithmic Intelligence to Linguistic Intelligence
Wang, F., Miao, Q., Li, X., Wang, X., & Lin, Y. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123486 · Cited by 214 · see citing works
Assessing The Factual Accuracy of Generated Text
Goodrich, B., Rao, V., Liu, P. J., & Saleh, M. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330955 · Cited by 162 · see citing works
Artificial intelligence hallucinations
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04473-y · Cited by 136 · see citing works
Factual Error Correction for Abstractive Summarization Models
Cao, M., Dong, Y., Wu, J., & Cheung, J. C. K. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.506 · Cited by 132 · see citing works
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Schuster, T., Fisch, A., & Barzilay, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.52 · Cited by 126 · see citing works
Detecting Hallucinated Content in Conditional Neural Sequence Generation
Zhou, C., Neubig, G., Gu, J., Diab, M., Guzmán, F., Zettlemoyer, L., & Ghazvininejad, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.120 · Cited by 122 · see citing works
Cognitive reflection, cognitive intelligence, and cognitive abilities: A meta-analysis
Otero, I., Salgado, J. F., & Moscoso, S. (2021) · Intelligence · doi:10.1016/j.intell.2021.101614 · Cited by 111 · see citing works
A Mathematical Investigation of Hallucination and Creativity in GPT Models
Lee, M. (2023) · Mathematics · doi:10.3390/math11102320 · Cited by 105 · see citing works
Reducing Quantity Hallucinations in Abstractive Summarization
Zhao, Z., Cohen, S. B., & Webber, B. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.203 · Cited by 99 · see citing works
Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding
Dziri, N., Madotto, A., Zaïane, O., & Bose, A. J. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.168 · Cited by 81 · see citing works
DAO to HANOI via DeSci: AI Paradigm Shifts from AlphaGo to ChatGPT
Miao, Q., Zheng, W., Lv, Y., Huang, M., Ding, W., & Wang, F.-Y. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123561 · Cited by 78 · see citing works
Masters, K. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2208731 · Cited by 76 · see citing works
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation
Liu, T., Zhang, Y., Brockett, C., Mao, Y., Sui, Z., Chen, W., & Dolan, B. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.464 · Cited by 63 · see citing works
Self-citationEvaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference
Dušek, O., & Kasner, Z. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.19 · Cited by 55 · see citing works
Focus Attention: Promoting Faithfulness and Diversity in Summarization
Aralikatte, R., Narayan, S., Maynez, J., Rothe, S., & McDonald, R. (2021) · doi:10.18653/v1/2021.acl-long.474 · Cited by 38 · see citing works
McIlroy-Young, R., Kleinberg, J. M., Sen, S., Barocas, S., & Anderson, A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534177 · Cited by 24 · see citing works
Identifying Fluently Inadequate Output in Neural and Statistical Machine Translation.
Martindale, M. J., Carpuat, M. J., Duh, K., & McNamee, P. (2019) · Cited by 22 · see citing works
Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View
Liu, T., Zheng, X., Chang, B., & Sui, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17583 · Cited by 21 · see citing works
Self-citationHeidenreich, H. S., & Williams, J. R. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462578 · Cited by 10 · see citing works
‘An Infinite Deal of Nothing’: critical ruminations on ChatGPT and the politics of language
Roy, N., & Maity, M. (2023) · DECISION · doi:10.1007/s40622-023-00342-3 · Cited by 6 · see citing works
Watters, P., & Patel, M. (1999) · PubMed · Cited by 5 · see citing works
Self-citationThapa, N., Abidi, A. I., & Sunil Kumar, G. (2022) · International Conference on Computing for Sustainable Global Development · doi:10.23919/indiacom54597.2022.9763225 · Cited by 3 · see citing works
Identifying Untrustworthy Samples
Shen, L., Zhan, H., Shen, X., Chen, H., Zhao, X., & Zhu, X. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482352 · Cited by 2 · see citing works
(2021) · The Research Repository @ WVU (West Virginia University)
Flagged first, then by citations
1 of the 49 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Game changers: A generative AI prompt protocol to enhance human-AI knowledge co-construction
Robertson, J., Ferreira, C. C., Botha, E., & Oosthuizen, K. (2024) · Business Horizons · doi:10.1016/j.bushor.2024.04.008 · Cited by 149 · see citing works
Generative AI Prompt Engineering for Educators: Practical Strategies
Park, J., & Choo, S. (2024) · Journal of Special Education Technology · doi:10.1177/01626434241298954 · Cited by 133 · see citing works
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works
“This technology has not only revolutionized the industry and academia, but has also reignited critical discussions concerning AI consciousness and its potential threats to humanity [9], [10], [11].”
BackgroundSelf-citationA Systematic Review of the Limitations and Associated Opportunities of ChatGPT
Cong‐Lem, N., Soyoof, A., & Tsering, D. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2344142 · Cited by 125 · see citing works
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · see citing works
“This lack of consideration results in benchmarks that wrongfully assume a uniform cognitive framework, failing to capture how LLMs handle linguistic variations across different cultures and languages [13].”
BackgroundInfluentialSelf-citationMcIntosh, T. R., Sušnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., Nowrozy, R., & Halgamuge, M. N. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103964 · Cited by 124 · see citing works
Self-citationAutomating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning
Sušnjak, T., Hwang, P., Reyes, N. H., Barczak, A. L. C., McIntosh, T. R., & Ranathunga, S. (2025) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3715964 · Cited by 91 · see citing works
Self-citationInteligencia artificial generativa y educación
García‐Peñalvo, F. J. (2024) · Education in the Knowledge Society (EKS) · doi:10.14201/eks.31942 · Cited by 128 · see citing works
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., & Halgamuge, M. N. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3377445 · Cited by 62 · see citing works
Self-citationIdentity, crimes, and law enforcement in the Metaverse
Qin, H. X., Wang, Y., & Hui, P. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-04266-w · Cited by 84 · see citing works
Generative AI in education: ChatGPT-4 in evaluating students’ written responses
Jauhiainen, J. S., & Guerra, A. G. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2024.2422337 · Cited by 63 · see citing works
Park, S., & Nan, X. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02620-3 · Cited by 58 · see citing works
Fake News Detection with Large Language Models on the LIAR DatasetPreprint
Boissonneault, D., & Hensen, E. (2024) · Research Square · doi:10.21203/rs.3.rs-4465815/v1 · Cited by 49 · see citing works
Microsoft Copilot and Anthropic Claude AI in education and library service
Adetayo, A. J., Aborisade, M. O., & Sanni, B. A. (2024) · Library Hi Tech News · doi:10.1108/lhtn-01-2024-0002 · Cited by 45 · see citing works
Swindell, A., Greeley, L., Farag, A., & Verdone, B. (2024) · Online Learning · doi:10.24059/olj.v28i2.4438 · Cited by 45 · see citing works
A Reasoning and Value Alignment Test to Assess Advanced GPT Reasoning
Mcintosh, T. R., Liu, T., Susnjak, T., Watters, P., & Halgamuge, M. N. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3670691 · Cited by 42 · see citing works
Self-citationMcIntosh, T. R., Sušnjak, T., Liu, T., Watters, P., Ng, A., & Halgamuge, M. N. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3394392 · Cited by 38 · see citing works
Self-citationA novel large-language-model-driven framework for named entity recognition
Wang, Z., Chen, H., Xu, G., & Ren, M. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104054 · Cited by 37 · see citing works
Enhancing Accuracy in Large Language Models Through Dynamic Real-Time Information InjectionPreprint
Ouyang, Q., Wang, S., & Wang, B. (2023) · Preprints.org · doi:10.20944/preprints202312.1987.v1 · Cited by 35 · see citing works
Lee, Y. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-13129-5 · Cited by 40 · see citing works
Jin, L., Shen, Z., Alhur, A., & Naeem, S. B. (2025) · Information Development · doi:10.1177/02666669251340954 · Cited by 38 · see citing works
McDonald, D., Papadopoulos, R., & Benningfield, L. (2024) · doi:10.36227/techrxiv.171665607.76504195/v1 · Cited by 34 · see citing works
Yang, X., Wang, Z., Wang, Q., Ke, W., Zhang, K., & Shi, J. (2024) · International Journal of Web Information Systems · doi:10.1108/ijwis-12-2023-0256 · Cited by 31 · see citing works
Franc, J. M., Hertelendy, A. J., Cheng, L. T. W., Hata, R., & Verde, M. (2024) · Journal of Medical Internet Research · doi:10.2196/55648 · Cited by 30 · see citing works
Kirchenbauer, J., & Barns, C. (2024) · doi:10.31219/osf.io/pv7r5 · Cited by 30 · see citing works
VaxBot-HPV: a GPT-based chatbot for answering HPV vaccine-related questions
Li, Y., Li, J., Li, M., Yu, E., Rhee, D., Amith, M. “., Tang, L., Savas, L. S., Cui, L., & Tao, C. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooaf005 · Cited by 29 · see citing works
Kumamoto, T., Yoshida, Y., & Fujima, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3719038/v1 · Cited by 28 · see citing works
Salminen, J. O., Jung, S., Medina, J., Aldous, K. K., Azem, J. Y., Akhtar, W., & Jansen, B. J. J. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3664690 · Cited by 27 · see citing works
Analyzing and Mitigating Cultural Hallucinations of Commercial Language Models in TurkishPreprint
Boztemir, Y., & Çalışkan, N. (2024) · doi:10.36227/techrxiv.171504678.80895811/v1 · Cited by 26 · see citing works
Kaate, I., Salminen, J. O., Jung, S., Xuan, T. T. T., Häyhänen, E., Azem, J. Y., & Jansen, B. J. J. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712160 · Cited by 29 · see citing works
Cost-efficient prompt engineering for unsupervised entity resolution in the product matching domain
Nananukul, N., Sisaengsuwanchai, K., & Kejriwal, M. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00159-8 · Cited by 23 · see citing works
Efficient Ransomware Detection via Portable Executable File Image Analysis By LLaMA-7bPreprint
Li, X., Zhu, T., & Zhang, W. (2023) · Research Square · doi:10.21203/rs.3.rs-3679775/v1 · Cited by 22 · see citing works
Wong, S. M., Leung, H.-F., & Wong, K. Y. (2024) · Research Square · doi:10.21203/rs.3.rs-4063228/v1 · Cited by 23 · see citing works
Mardiansyah, K., & Surya, W. (2024) · Research Square · doi:10.21203/rs.3.rs-4005702/v1 · Cited by 23 · see citing works
McPhee, S., & Jerowsky, M. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1593278 · Cited by 32 · see citing works
Reducing Hallucinations in Large Language Models Through Contextual Position EncodingPreprint
Desrochers, S., Wilson, J., & Beauchesne, M. (2024) · doi:10.31219/osf.io/exjqb · Cited by 21 · see citing works
A Comparative Analysis to Evaluate Bias and Fairness Across Large Language Models with BenchmarksPreprint
陳文意, & 黃兆明 (2024) · doi:10.31219/osf.io/mc762 · Cited by 19 · see citing works
盖振华, Tong, L., & Ge, Q. (2024) · doi:10.31219/osf.io/ctw8v · Cited by 19 · see citing works
Birillo, A., Artser, E., Potriasaeva, A., Vlasov, I., Dzialets, K., Golubev, Y., Герасимов, И. В., Keuning, H., & Bryksin, T. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699556 · Cited by 23 · see citing works
MisCaltral: Reducing Numeric Hallucinations of Mistral with Precision Numeric CalculationPreprint
Chen, S.-W., & Hsu, H.-J. (2023) · Research Square · doi:10.21203/rs.3.rs-3789011/v1 · Cited by 17 · see citing works
Implementing Automated Error Correction and Feedback Loops in Kimi, A Chinese Large Language ModelPreprint
Cheung, W.-L., & Luk, C.-Y. (2024) · doi:10.31219/osf.io/7vpxr · Cited by 17 · see citing works
Combining LoRA to GPT-Neo to Reduce Large Language Model HallucinationPreprint
Huang, S.-H., & Chen, C.-Y. (2024) · Research Square · doi:10.21203/rs.3.rs-4515250/v1 · Cited by 17 · see citing works
Integrating Deep Learning with Symbolic Reasoning in TinyLlama for Accurate Information RetrievalPreprint
Xiong, X., & Zheng, M. (2024) · Research Square · doi:10.21203/rs.3.rs-3883562/v1 · Cited by 17 · see citing works
Tsai, H.-C., Huang, Y.-F., & Kuo, C.-W. (2024) · Research Square · doi:10.21203/rs.3.rs-4022248/v1 · Cited by 17 · see citing works
Kanato, S. A. T. O., Kaneko, H., & Fujimura, M. (2024) · doi:10.31219/osf.io/4hzya · Cited by 16 · see citing works
Does Social Bot Help Socialize? Evidence from a Microblogging Platform
Gao, Y. A., Zhang, M. M., & Lysyakov, M. (2025) · Information Systems Research · doi:10.1287/isre.2024.1089 · Cited by 14 · see citing works
Semantic Analysis of Phishing Emails Leading to Ransomware with ChatGPTPreprint
Fujima, H., Takeuchi, K., & Kumamoto, T. (2023) · Research Square · doi:10.21203/rs.3.rs-3670198/v1 · Cited by 14 · see citing works
RansomCillin: Leveraging NTFS Spare Space to Recover from Ransomware AttacksPreprint
Takeuchi, K., Fujima, H., Kumamoto, T., & Yoshida, Y. (2023) · doi:10.36227/techrxiv.24596754.v1 · Cited by 14 · see citing works
Zhang, Y., & Chen, X. (2023) · Research Square · doi:10.21203/rs.3.rs-3778538/v1 · Cited by 14 · see citing works
Dynamic Supplementation of Federated Search Results for Reducing Hallucinations in LLMsPreprint
Chen, J., Huang, X., & Li, Y. (2024) · doi:10.31219/osf.io/x5vge · Cited by 14 · see citing works
Knowledge Accuracy and Reducing Hallucinations in LLMs via Dynamic Domain Knowledge InjectionPreprint
Capellini, R., Atienza, F., & Sconfield, M. (2024) · Research Square · doi:10.21203/rs.3.rs-4540506/v1 · Cited by 14 · see citing works
A Comparative Analysis of Cultural Alignment in Large Language Models in Bilingual ContextsPreprint
Yuan, X., Hu, J., & Zhang, Q. (2024) · doi:10.31219/osf.io/6hpcf · Cited by 14 · see citing works
Optimizing Machine Translation: A Benchmarking Suite for Efficiency and Quality Enhancement
Kamuni, N., Dodda, S., Chintala, S., & Kunchakuri, N. (2024) · International Conference on Computing, Sciences and Communications (ICCSC) · doi:10.1109/iccsc62048.2024.10830335 · Cited by 13 · see citing works
The citation catastrophe: Propagation of AI-generated counterfeit citations in scholarship
Camp, N. T., Bengtson, J. A., & Sandstrom, J. C. (2025) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2025.103065 · Cited by 12 · see citing works
Assessing the Impact of GPT-4 Turbo in Generating Defeaters for Assurance Cases
Shahandashti, K. K., Sivakumar, M., Mohajer, M. M., Belle, A. B., Wang, S., & Lethbridge, T. C. (2024) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3650105.3652291 · Cited by 12 · see citing works
Chen, J., Lu, S., & Zhong, L. (2024) · Applied Sciences · doi:10.3390/app14177716 · Cited by 12 · see citing works
Unmoderated Usability Studies Evolved: Can GPT Ask Useful Follow-up Questions?
Kuric, E., Demcak, P., & Krajcovic, M. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2427978 · Cited by 12 · see citing works
Cross-Cultural Implications of Large Language Models: An Extended Comparative Analysis
Shan, X., Xu, Y., Wang, Y., Lin, Y.-S., & Bao, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-76806-4_8 · Cited by 12 · see citing works
Adaptive Query Contextualization Algorithm for Enhanced Information Retrieval in Alpaca LLMPreprint
Kuo, C.-W., Huang, Y.-F., & Tsai, H.-C. (2023) · Research Square · doi:10.21203/rs.3.rs-3806145/v1 · Cited by 12 · see citing works
Shadiev, R., Nguyen, T. M., & Hwang, W. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2025.2612252 · Cited by 14 · see citing works
Analysing the Role of Generative AI in Software Engineering - Results from an MLR
Bazzan, T., Olojo, B., Majda, P., Kelly, T., Yılmaz, M., Marks, G., & Clarke, P. M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-71139-8_11 · Cited by 13 · see citing works
Using ChatGPT to Analyze Ransomware Messages and to Predict Ransomware ThreatsPreprint
Fujima, H., Kumamoto, T., & Yoshida, Y. (2023) · Research Square · doi:10.21203/rs.3.rs-3645967/v1 · Cited by 11 · see citing works
Measuring the Visual Hallucination in ChatGPT on Visually Deceptive ImagesPreprint
Ping, L., Gu, Y., & Feng, L. (2024) · doi:10.31219/osf.io/v23fr · Cited by 11 · see citing works
Kaate, I., Salminen, J. O., Jung, S., Santos, J. M., Häyhänen, E., Xuan, T. T. T., Azem, J. Y., & Jansen, B. J. J. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2469659 · Cited by 10 · see citing works
Investigating Hallucination Tendencies of Large Language Models in Japanese and EnglishPreprint
Tsuruta, H., & Sakaguchi, R. (2024) · Research Square · doi:10.21203/rs.3.rs-4521710/v1 · Cited by 10 · see citing works
Salminen, J. O., Jung, S., Medina, J., Aldous, K. K., Azem, J. Y., Akhtar, W., Häyhänen, E., & Jansen, B. J. J. (2024) · International Academic Mindtrek Conference (Mindtrek) · doi:10.1145/3681716.3681727 · Cited by 9 · see citing works
Wei, X. (2025) · Theoretical and Natural Science · doi:10.54254/2753-8818/2025.20110 · Cited by 9 · see citing works
Nowrozy, R., Ahmed, K. E. U., & Wang, H. (2025) · PLoS ONE · doi:10.1371/journal.pone.0310553 · Cited by 12 · see citing works
Su, X., & Gu, Y. (2024) · doi:10.31219/osf.io/ns2v3 · Cited by 9 · see citing works
VaxBot-HPV: A GPT-based Chatbot for Answering HPV Vaccine-related QuestionsPreprint
Tao, C., Li, Y., Li, J., Li, M., Yu, E., Amith, M. “., Tang, L., Savas, L. S., & Cui, L. (2024) · Research Square · doi:10.21203/rs.3.rs-4876692/v1 · Cited by 9 · see citing works
Bane and boon of hallucinations in context of generative AIPreprint
Patel, H. (2024) · doi:10.36227/techrxiv.171198062.20183635/v1 · Cited by 10 · see citing works
Ho-tin, W., & Yip, G.-L. (2024) · doi:10.31219/osf.io/wkb32 · Cited by 8 · see citing works
Zhao, J., Huang, C., & Li, X. (2024) · Research Square · doi:10.21203/rs.3.rs-4566507/v1 · Cited by 8 · see citing works
Mitigating Hallucinations in Large Language Models with Sliding Generation and Self-ChecksPreprint
Harrington, F., Rosenthal, E., & Swinburne, M. (2024) · doi:10.36227/techrxiv.172297712.23792600/v1 · Cited by 8 · see citing works
Automated Methodologies for Evaluating Lying, Hallucinations, and Bias in Large Language ModelsPreprint
Ecurali, G., & Thackeray, Z. (2024) · Research Square · doi:10.21203/rs.3.rs-4855434/v1 · Cited by 8 · see citing works
Liu, Z., Kou, J., Zhang, W., Gu, C., Fang, X., Huang, Z., Hao, Y., Li, H., Lu, X., Yin, A., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189137 · Cited by 9 · see citing works
Using GPT-4 Turbo to Automatically Identify Defeaters in Assurance Cases
Shahandashti, K. K., Belle, A. B., Mohajer, M. M., Odu, O., Lethbridge, T. C., Hemmati, H., & Wang, S. (2024) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew61692.2024.00011 · Cited by 7 · see citing works
LLaMALoop: Enhancing Information Retrieval in LLaMA with Semantic Relevance Feedback LoopPreprint
Tsai, H.-C., Kuo, C.-W., & Huang, Y.-F. (2023) · Research Square · doi:10.21203/rs.3.rs-3810235/v1 · Cited by 7 · see citing works
Li, P.-H., & Lai, Y.-Y. (2024) · doi:10.31219/osf.io/ma6cq · Cited by 7 · see citing works
Detecting LLM Hallucinations Using Monte Carlo Simulations on Token ProbabilitiesPreprint
Ledger, G., & Mancinni, R. (2024) · doi:10.36227/techrxiv.171822396.61518693/v1 · Cited by 7 · see citing works
Boosting Long-term Factuality in Large Language Model with Real-World Entity QueriesPreprint
Davies, L., & Bellington, S. (2024) · Research Square · doi:10.21203/rs.3.rs-4834886/v1 · Cited by 7 · see citing works
Nadeem, M., Sohail, S. S., Madsen, D. Ø., Alzahrani, A. I., Ser, J. D., & Muhammad, K. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3536937 · Cited by 6 · see citing works
LENS: Layers of Evaluation of Hallucination in GenAI Systems
Gu, C., Zhang, W., Huang, Z., Kou, J., Liu, Z., Zhao, C., Liu, C., Zhang, L., Lin, W., Wang, Z., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189150 · Cited by 16 · see citing works
AI Asyet Another Tool in Undergraduate Student Projects: Preliminary Results
Pérez-Colado, I. J., Freire, M., Calvo-Morata, A., Pérez-Colado, V. M., & Fernández‐Manjón, B. (2024) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon60312.2024.10578883 · Cited by 6 · see citing works
Investigating the Utility of ChatGPT in the Issue Tracking System: An Exploratory Study
Das, J. K., Mondal, S., & Roy, C. K. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3645083 · Cited by 6 · see citing works
Reinforcement Learning for Optimized Information Retrieval in LLaMAPreprint
Tu, C.-H., Hsu, H.-J., & Chen, S.-W. (2024) · Research Square · doi:10.21203/rs.3.rs-3847100/v1 · Cited by 6 · see citing works
Gong, X., Liu, M., & Chen, X. (2024) · doi:10.31219/osf.io/srgpx · Cited by 6 · see citing works
Relational & culture-sensitive AI innovation
Herzog, C. (2025) · Journal of Responsible Technology · doi:10.1016/j.jrt.2025.100135 · Cited by 7 · see citing works
A systematically derived AI-based framework for student-centered learning in higher education
Albasry, H., Carmona-Cejudo, E., Rauf, A., & Chen, D. (2025) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2025.102085 · Cited by 5 · see citing works
Thao, N. T. P., Tam, D. H., Duc, M. L., & Sang, N. Q. (2025) · Sustainable Futures · doi:10.1016/j.sftr.2025.101523 · Cited by 5 · see citing works
Ethical Guidelines for the Application of Generative AI in German Journalism
Hofeditz, L., Jung, A.-K., Mirbabaie, M., & Stieglitz, S. (2025) · Digital Society · doi:10.1007/s44206-024-00151-w · Cited by 7 · see citing works
Implementing An Automated Socratic Method to Reduce Hallucinations in Large Language ModelsPreprint
Underwood, H., & Fenwick, Z. (2024) · doi:10.31219/osf.io/grh4b · Cited by 5 · see citing works
van Eerde, A. M., Teixeira, A., Galletti, F., Maternik, M., Capone, V. P., Westland, R., Mulder, J., Halbritter, J., Osterholt, T., Neukel, V., et al. (2025) · Pediatric Nephrology · doi:10.1007/s00467-025-06746-w · Cited by 4 · see citing works
ChatGPT and library users: AI risks of hallucinations and misinformation
Oladokun, B. D., Emmanuel, V. O., Osagie, O., & Alabi, A. O. (2025) · Cybrarians Journal · doi:10.70000/cj.2025.76.642 · Cited by 6 · see citing works
Using LLMs to Adapt Serious Games with Educators in the Loop
Bonetti, F., Bucchiarone, A., & Wanick, V. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78269-5_7 · Cited by 5 · see citing works
Opanasenko, Y., Bardone, E., Pedaste, M., & Siiman, L. A. (2024) · Education Sciences · doi:10.3390/educsci15010028 · Cited by 4 · see citing works
Anthony, A., Sharma, S., & Sharma, S. (2025) · Bulletin of Science Technology & Society · doi:10.1177/02704676251353105 · Cited by 8 · see citing works
RansomCillin: Leveraging NTFS Spare Space to Recover from Ransomware AttacksPreprint
Takeuchi, K., Fujima, H., Kumamoto, T., & Yoshida, Y. (2023) · Research Square · doi:10.21203/rs.3.rs-3648063/v1 · Cited by 4 · see citing works
The Double-Edged Sword of AI Writing ToolsPreprint
Xu, Y. (2024) · doi:10.31235/osf.io/wnpd5 · Cited by 4 · see citing works
Large Language Model Understands Chinese Better with Mega TokenizationPreprint
Lu, X., Wang, Q., & Liu, X. (2024) · Research Square · doi:10.21203/rs.3.rs-4542532/v1 · Cited by 4 · see citing works
- Authors
- McIntosh, T. R., Liu, T., Susnjak, T., Watters, P., Ng, A., & Halgamuge, M. N.
- Affiliations
- La Trobe University (AU)
- Massey University (NZ)
- Massey University, Auckland, New Zealand
- Cyberstronomy Pty Ltd, Ballarat, VIC, Australia
- RMIT University (AU)
- Venue
- IEEE Transactions on Artificial Intelligence
- Published
- 2024-06
- Volume
- 5
- Issue
- 6
- Pages
- 2739-2751
- ISSN
- 2691-4581 (ISSN-L)
- OpenAlex
- W4388691793
- Cited by
- 145
- References
- 49
- Citations by year
- 2026: 12 · 2025: 31 · 2024: 90 · 2023: 12
- FWCI
- 13.67
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Licence
- Locations
- IEEE Transactions on Artificial Intelligence (publishedVersion)
- Figshare (submittedVersion, cc-by)
- La Trobe University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- hallucination evaluation, ChatGPT-4, Google Bard, Perplexity AI, controversial issues, brain teasers, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
